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I.  INTRODUCTION 


A.  MOTIVATION 

Within  industry,  it  is  estimated  that  roughly  66%  of  a  business’s  information 
technology  (IT)  budget  is  spent  on  ongoing  operations  and  maintenance  and  not  on  new 
initiatives  and  projects  (“Enterprise  and  SMB  Software  Survey  North  America  and 
Europe,”  2008).  If  DoD’s  IT  operating  budget  follows  this  trend,  then  roughly  $20B 
(Lynn  III,  2009)  of  DoD’s  2010  IT  budget,  or  3.7%  of  DoD’s  total  2010  base  budget 
(“Defense  Budget  2010,”  2010),  could  be  spent  on  maintaining  the  status  quo.  To 
continue  to  support  the  warfighter  and  be  fiscally  responsible,  DoD  needs  to  eliminate  the 
procurement  and  maintenance  of  redundant  infrastructure. 

Currently,  DoD  acquires  hardware,  software,  and  personnel  that  in  essence 
perform  the  same  function.  The  waste  does  not  stop  there.  DoD  must  also  pay  for  the 
hardware  and  software  maintenance,  licensing,  facilities  requirements  (e.g.,  power, 
cooling,  network,  floor  space),  and  keep  personnel  on  staff  to  maintain  everything.  In 
September  2009,  Deputy  Secretary  of  Defense  William  Lynn  III  stated  that  within  DoD 
there  are  approximately  15,000  networks  comprised  of  7  million  computers,  laptops, 
servers,  and  other  devices.  It  takes  an  astounding  90,000  personnel  to  administer,  monitor 
and  defend  those  networks  (Lynn  III,  2009). 

In  June  2009,  a  memo  was  sent  from  the  Executive  Office  of  the  President  to  the 
heads  of  departments  and  agencies  regarding  fiscal  2011  budget  planning.  What  is 
interesting  about  this  memo  is  that  under  information  technology,  cloud  computing  is 
specifically  called  out  as  a  Presidential  priority  for  IT.  Specifically,  the  memo  states  that 
budget  IT  submissions  “...should  support  the  President’s  priorities  for  information 
technology,  including  transparency,  participation  and  collaboration,  and  improving 
innovation,  efficiency  and  effectiveness,  in  areas  like  cloud  computing...”  (Orszag,  2009) 
Within  the  President’s  2011  budget,  cloud  computing  is  highlighted  as  a  major  part  of  the 
strategy  to  achieve  efficient  and  effective  IT.  The  Office  of  Management  and  Budget 
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(OMB),  as  part  of  the  FY  2011  budget  process,  has  requested  all  agencies  evaluate  cloud 
computing  alternatives  as  part  of  their  budget  submissions  for  all  relevant  major  IT 
investments.  In  FY  2012,  agencies  will  be  expected  to  tell  OMB  why  they  cannot  use 
cloud  computing  for  any  new  major  technology  project.  In  FY  2013,  agencies  must  give 
OMB  a  complete  alternative  analysis  for  how  existing  projects  could  be  moved  to  cloud 
computing. 

Specifically: 

■  By  September  2011  -  all  newly  planned  or  performing  major  IT 
investments  acquisitions  must  complete  an  alternatives  analysis  that 
includes  a  cloud  computing  based  alternative  as  part  of  their  budget 
submissions 

■  By  September  2012  -  all  IT  investment  making  enhancements  to  an 
existing  investment  must  complete  an  alternatives  analysis  that  includes  a 
cloud  computing  based  alternative  as  part  of  their  budget  submissions. 
This  includes  any  project  where  funding  is  used  for  development, 
modernization,  enhancement,  or  simply  operations  and  maintenance 
(Miller,  2009) 

■  By  September  2013  -  all  IT  investments  in  steady-state  must  complete  an 
alternatives  analysis  that  includes  a  cloud  computing  based  alternative  as 
part  of  their  budget  submissions  (Kundra,  2010) 

While  cloud  computing  is  not  a  mandate,  when  the  President  asks  you  to  consider 
something  people  tend  to  listen.  Cloud  computing  holds  promise  to  reduce  IT 
infrastructure  needs — both  up-front  and  support  costs,  decrease  maintenance/upgrades, 
improve  resource  utilization  and  collaboration  capabilities.  Although  cloud-computing 
products  are  available  these  products  are  not  mature  enough  to  deliver  the  quality-of- 
service  required  by  DoD. 

In  this  thesis,  we  identify  the  requirements  for  cloud  computing  to  support  DoD’s 
unclassified  NIPRNet  computing  needs  with  a  specific  focus  on  workflow  processes 
within  the  Army  Test  and  Evaluation  (T&E)  domain.  The  U.S.  Army  was  chosen  because 
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it  is  the  largest  branch  of  DoD  (“Bureau  of  Labor  Statistics:  Career  Guide  to  Industries,” 
2010)  and  because  the  Army  is  pushing  forward  with  plans  to  deploy  an  Army  private 
cloud  computing  (APC2)  environment.  The  request  for  proposal  (RFP)  for  APC2  was 
released  in  late  July  2010  with  the  stated  goal  of  reducing  cost  and  energy  use  while 
improving  the  Army’s  cyber-security  posture  and  speed  of  innovation  (Army  Contracting 
Command,  2010).  The  APC2  is  intended  to  be  the  cornerstone  of  a  broader  data  center 
consolidation  initiative  that  aims  to  reduce  and  consolidate  the  number  of  Army  data 
centers  to  less  than  twenty  (Hoover,  2010b). 

In  its  present  form,  the  Army  has  Network  Enterprise  Centers  (NEC)  located  at 
447  locations  in  the  United  States  supporting  nineteen  different  commands  and  agencies. 
The  Anny  CIO  has  made  it  a  top  priority  to  realign  and  consolidate  these  NECs  into  two 
Network  Service  Centers  (NSC)  located  in  the  United  States  and  three  abroad.  At  the 
heart  of  each  NSC  will  be  an  Area  Processing  Center  (APC),  or  consolidated  data  center 
(Sean  Gallagher,  2010).  APCs  provide  theater-level  IT  capabilities  where  functional  and 
common-services  information  is  stored,  replicated,  and  centrally  managed.  The  goal  of  an 
APC  is  to  pull  all  applications  and  data  storage  out  of  local  data  centers  at  Army  facilities 
and  centralize  it  at  a  regional  data  center  with  applications  mirrored  across  each  APC. 
Data  center  consolidation  is  about  determining  how  to  collapse  and  provide  more  shared 
services,  which  also  is  a  key  step  in  adoption  of  the  cloud  (Link,  2010). 

With  APC2,  the  Anny  will  be  converting  designated  APCs  into  cloud  computing 
environments  that  can  provide  shared  services.  The  latest  round  of  APCs  being 
established  includes  Redstone  Arsenal  (RSA),  Fort  Knox,  and  Fort  Bragg  (Corrin,  2010). 
Since  an  Army  T&E  Command  (ATEC)  Developmental  Test  Center  (DTC)  subordinate 
command,  Redstone  Test  Center  (RTC),  is  located  on  RSA  this  research  will  approach 
the  subject  of  a  private  DoD  cloud  from  an  Army  T&E  perspective.  We  assume  that 
being  collocated  with  an  APC  (i.e.,  utilizing  the  same  fiber  backbone)  will  provide  the 
lowest  possible  latencies  available  in  a  cloud  environment  and  as  such  would  provide  a 
best-case  scenario  for  testing  workflow  processes. 
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B.  RESEARCH  QUESTIONS 

This  research  attempts  to  answer  the  following  questions: 

1)  What  communications/collaborations  do  users  use  their  computers  for  in 
the  execution  of  a  typical  Army  T&E  program? 

2)  Can  these  interactions  be  abstractly  modeled? 

3)  How  would  these  collaborations/communications  be  carried  out  in  a  cloud 
based  environment? 

C.  BENEFITS  OF  STUDY 

This  research  will  assist  DoD  in  defining  a  roadmap  for  addressing  the 
Presidential  IT  priority.  This  will  be  accomplished  by: 

•  Providing  a  general  overview  of  the  current  state  of  cloud  computing 

•  Identifying  issues  that  should  be  addressed  before  attempting  to  move  to  a 
cloud  computing  environment 

•  Analysis  of  the  feasibility  of  a  cloud  environment  in  meeting  the  Anny 
T&E  mission  through  multiple  use  cases 

D.  ORGANIZATION 

Subsequent  chapters  will  focus  on  cloud  computing  concepts,  as-is  use  cases,  the 
study’s  results,  and  final  conclusions. 

Below  are  synopses  of  each  chapter’s  contents: 

1.  Chapter  II:  Background 

In  order  to  have  a  firm  foundation  for  the  topics  discussed,  and  a  common 
lexicon,  Chapter  II  provides  a  broad  overview  on  cloud  computing.  The  historical 
background  leading  up  to  the  current  cloud  computing  hype  will  be  discussed.  A  brief 
definition  and  description  of  the  core  concepts  of  cloud  computing  will  be  covered  and  a 
brief  overview  of  current  efforts  within  the  federal  government  will  be  given.  The 
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overview  of  cloud  computing  will  include  discussions  on:  essential  characteristics, 
architectures,  deployment  models,  underlying  value,  and  the  identification  of  concerns 
that  should  be  evaluated  before  moving  to  a  cloud  environment. 

2.  Chapter  III:  Current  T&E  Process 

Chapter  III  presents  several  as-is  use  cases  describing  processes  that  could 
potentially  be  improved  within  a  cloud  environment.  The  scope  for  this  chapter  will  be 
limited  to  the  DoD  Army  T&E  enterprise  domain.  The  T&E  domain  will  be  used  as  an 
example  to  contrast  the  current  as-is  solution,  as  described  in  Chapter  III,  with  a  potential 
solution  provided  by  a  private  DoD  cloud,  as  described  in  Chapter  IV.  This  chapter  will 
provide  a  brief  description  of  select  systems  within  the  Army  T&E  Enterprise 
Architecture.  Use  cases  will  be  generated  and  a  common  high-level  mission  scenario  will 
be  used  to  walk  through  the  current  project  management,  range -test  scheduling,  and  test¬ 
reporting  process  within  ATEC.  The  selected  use  cases  were  chosen  as  they  can  readily 
be  extrapolated  to  other  domains  within  DoD.  Every  DoD  activity  also  has  project 
management,  report  collaboration,  and  asset  utilization/scheduling  concerns.  Use  cases, 
process  diagrams,  and  collaboration  diagrams  are  used  in  this  thesis  to  model  ATEC’s 
T&E  workflow  processes. 

3.  Chapter  IV:  Cloud  Based  T&E  Process 

Chapter  IV  documents  how  a  subset  of  the  process,  described  within  Chapter  III, 
could  be  modified  to  leverage  cloud  computing.  This  chapter  provides  an  assessment  of 
how  the  mission  scenario  previously  presented  could  be  accomplished  within  a  private 
DoD  cloud.  A  subset  of  the  use  cases,  process,  and  collaboration  diagrams  from  Chapter 
III  will  be  assessed  to  see  how  cloud  computing,  in  its  current  fonn,  could  potentially 
streamline  existing  processes  while  also  providing  additional  functionality,  visibility,  and 
reducing  latency  in  delivering  information  to  senior  leadership. 


5 


4.  Chapter  V:  Conclusions  and  Future  Research 

Chapter  V  will  summarize  the  research  and  reiterate  what  has  been  learned  about 
cloud  computing  and  its  potential  applicability  to  DoD,  specifically  Army  T&E.  It  also 
contains  recommendations  on  how  to  proceed  and  identifies  follow-on  research  that 
should  be  conducted  but  was  outside  the  scope  of  this  thesis.  It  also  contains  potential 
concerns,  technical  challenges,  and  cultural  change  that  must  be  overcome  before 
widespread  adoption  of  a  cloud  environment  will  occur  within  DoD.  The  findings  of  this 
thesis  combined  with  follow-on  research  will  provide  DoD  a  more  comprehensive 
understanding  of  how  cloud  computing  environments  can  be  leveraged  to  improve  T&E 
operations,  streamline  processes,  and  reduce  cost. 

E.  KEY  FINDINGS  AND  RECOMMENDATIONS 

In  this  thesis,  we  analyzed  existing  processes  that  Army  T&E  users  follow  during 
the  execution  of  a  typical  T&E  program.  We  then  assessed  how  cloud  computing  can  be 
used  to  streamline  these  workflows. 

In  the  course  of  documenting  the  Army  T&E  workflow  processes,  we  focused  our 
attention  on  communications  and  collaborations  within  the  enterprise.  This  included 
communications  starting  with  a  request  for  test  services,  followed  by  scheduling  a  test, 
and  compilation  and  delivery  of  a  final  test  report  (TR).  The  documentation  consisted  of 
use  cases,  activity  diagrams,  and  collaboration  diagrams  for  nine  different  scenarios. 
During  the  analysis  of  the  current  system,  we  determined  that  the  current  system  relies 
heavily  on  e-mail  and  manual  processes  as  the  primary  means  of  communication  and  file 
transport. 

Currently,  information  is  relayed  in  a  very  serial  manner,  with  additional  delay 
introduced  whenever  someone  in  the  chain  of  communication  is  unavailable.  Even 
though  the  information  being  requesting,  in  most  cases,  is  available  locally  to  the  Test 
Center  (TC)  Test  Engineer  (TE)  or  Resource  Manager  (RM),  ATEC  does  not  have  access 
to  the  information.  In  other  words,  storing  the  data  locally,  that  is  on  the  workstation,  is 
an  impediment  to  information  sharing  within  ATEC  and  with  ATEC’s  stakeholders. 
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Improvements  could  be  obtained  through  the  creation  of  a  cloud-based  integrated 
working  environment  (IWE),  or  “one-stop-shop,”  where  infonnation  could  be  relayed  to 
Program  Managers  (PM)  or  other  ATEC  stakeholders  in  a  more  efficient  manner. 

The  workflows  could  be  streamlined  through  the  use  of  cloud-based  collaboration 
tools,  such  as:  online  document  editors,  instant  messaging,  threaded  message  boards, 
wikis,  blogs,  tags,  status  updates,  news,  hot  topics,  tasks,  and  RSS  feeds.  These 
collaboration  tools,  along  with  all  collected  data  associated  with  a  program,  would  be 
accessible  through  the  IWE.  The  IWE  would  allow  anyone  with  proper  access  rights  to 
pull  infonnation  relating  to  programs  of  interest  from  any  location  at  any  time,  cross- 
platfonn  cross-device,  and  would  take  the  middle -man,  the  human-in-the-loop,  out  of  the 
process.  Using  the  IWE  as  the  primary  mechanism  for  collaboration  would  also  help  DoD 
amass  the  large  amount  of  undocumented  corporate  knowledge  employees  currently 
posses,  in  their  heads,  into  documented  and  searchable  data. 

While  cloud  computing  is  still  in  its  infancy,  this  research  has  shown  that  it  does 
bear  promise  to  cut  the  cost  of  delivering  IT  services  to  the  DoD  community,  other  things 
being  equal.  Cloud  computing  is  a  disruptive  technology  whose  implementation  will 
require  change  across  all  levels  of  DoD.  It  will  require  technical  training  and  a  cultural 
shift  in  how  DoD  senior  leadership,  program  management,  end  users,  customers, 
suppliers,  and  especially  IT  professionals  think  about  IT  resources.  This  shift  will  require 
changes  in  all  aspects  of  the  acquisition  of  IT.  In  addition  to  the  technical  challenges, 
there  will  be  challenges  in  aligning  the  corporate  culture  with  the  new  workflows  and 
associated  means  of  communication  and  collaboration. 
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II.  BACKGROUND 


A.  CLOUD  COMPUTING 
1.  Background 


Figure  1.  Typical  Network  Diagram  from  (Jeffrey,  2009) 


For  years,  network  diagrams  have  used  a  cumulus  cloud  to  represent  the  Internet. 
The  cloud  image  indicated  something  vague,  intangible,  but  still  necessary  to  include  in 
the  diagram.  Lines  on  the  network  did  nothing  but  travel  through  the  cloud,  indicating 
data  passing  over  the  Internet  and  no  one  cared  where  the  messages  went.  On  security- 
focused  diagrams,  the  line  through  the  cloud  might  include  a  padlock  beside  it  to  indicate 
that  the  connection  is  secure.  The  cloud  has  now  been  promoted  to  a  first-class  actor  in 
the  network  diagram  itself.  Rather  than  simply  passing  through  the  cloud,  lines  now 
connect  to  the  cloud  and  use  it  as  part  of  an  application  (O’Neill,  2009).  Infonnation, 
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files,  and  applications  that  historically  have  been  stored  either  on  a  user’s  machine  or  a 
local  server  are  now  being  created,  stored,  manipulated,  and  accessed  through  the  cloud. 
This  makes  the  information  accessible  from  anywhere  and  allows  for  the  viewing  of  data 
in  many  different  ways. 


Servers,  Hmail,  j 

Networks,  J  'CalsFKjarinH 


ll> 


Software, 
Applications. 


Web  J.O: 
You  Tube, 
Hacebook, 
I'wttlBf 


Figure  2.  Cloud  Computing  Network  Diagram  from  (“DOE  Deploys  Cloud 

Computing,”  2010) 


The  model  behind  cloud  computing  is  not  a  new  or  revolutionary  concept.  In  the 
1960s,  John  McCarthy  theorized  that  “ computation  may  someday  be  organized  as  a 
public  utility .”  (Warsh,  2006)  However,  only  within  the  last  few  years  have  enough 
elements  come  together  to  create  a  perfect  storm  that  has  provided  an  environment  in 
which  cloud  computing  can  flourish  (e.g.,  technological  advances,  business  case,  Web 
2.0,  and  economic  uncertainty). 

For  many  years,  the  major  hurdle  for  cloud  computing  was  related  to  the  network: 
the  available  network  bandwidth  just  did  not  make  large  scale  cloud  computing  a  viable 
technical  solution.  However,  the  dotcom  boom  of  the  1990s,  and  resulting  exponential 
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growth  of  the  size  of  the  Internet,  set  the  stage  to  change  this.  The  dotcom  boom  brought 
with  it  a  massive  investment  in  fiber  optic  and  high  bandwidth  infrastructure, 
subsequently  raising  the  percentage  of  the  population  with  high-speed  access  to  the 
Internet. 

The  business  model  for  cloud  computing  was  established  after  the  dotcom  crash 
of  2000.  Prior  to  the  crash,  venture  capital  for  startup  businesses  could  easily  be  found 
because  everyone  wanted  to  cash  in  on  the  “new  industrial  revolution”  (Authers  & 
Mackenzie,  2010).  However,  after  the  stock  market  crashed,  capital  for  startup  businesses 
quickly  dried  up  and  Internet  companies  had  to  actually  have  a  business  plan.  This  left 
new  startups  with  a  tough  choice.  Startups  could  purchase  servers,  software,  licenses,  etc. 
that  would  enable  their  business  to  merely  get  off  the  ground  with  little  excess 
capabilities  for  growth,  or  invest  a  large  amount  of  startup  capital  in  capabilities  that  may 
never  be  used.  The  former  would  work  great,  if  the  business  slowly  gained  popularity  and 
allowed  for  the  ordering  and  integration  of  new  hardware.  However,  if  popularity  quickly 
grew,  traffic  could  overwhelm  the  baseline  servers  leading  to  customer  dissatisfaction 
with  the  quality  of  service  and  potentially  the  downfall  of  the  business.  So,  what  is  a 
startup  company  to  do?  Meanwhile,  Internet  companies  like  Amazon™  and  Google™ 
were  amassing  great  numbers  of  servers  in  large-scale  datacenters  to  meet  their 
businesses’  peak  usage  demands:  Amazon,  to  meet  its  ever  growing  presence  in  e- 
commerce  and  the  associated  peaks,  such  as  the  day  after  Thanksgiving,  and  Google  to 
index  the  ever  expanding  Internet,  while  providing  search  results  faster  than  its 
competitors.  However,  having  enough  physical  hardware  on  hand  to  meet  the  peak  usage 
meant  that  for  the  majority  of  the  time  this  hardware  would  be  grossly  underutilized. 
Regardless  of  the  utilization  levels,  Amazon  and  Google  had  to  pay  to  house  the 
hardware,  power  and  cool  it,  and  employ  an  army  of  technicians  to  maintain 
everything — all  resulting  in  a  large  overhead  operating  expense  for  both  companies. 
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Figure  3.  Large  Data  Center  from  (“Computer  History,”  2010) 


By  2006,  Amazon  was  looking  at  its  excess  computing  cycles  not  as  a  fiscal  drain 
on  the  company  but  rather  as  a  business  opportunity.  Since  these  huge  datacenters  already 
had  high-speed  connections  into  the  Internet  hubs,  Amazon  began  renting  usage  of  its 
datacenter’s  spare  computing  cycles.  The  offerings  were  quickly  utilized  by  small 
companies,  especially  startup  businesses,  as  this  partially  addressed  the  dilemma  of 
having  to  decide  how  much  hardware  to  purchase,  and  gave  startup  companies  the 
opportunity:  to  access  hardware  in  amounts  that  these  companies  would  have  otherwise 
been  unable  to  access,  only  paying  for  what  they  use.  Amazon  and  Google’s  entrance  into 
cloud  computing,  specifically  Infrastructure  as  a  Service  (IaaS),  has  been  compared 
magnitude  wise  to  the  tumultuous  change  that  the  electric  industry  went  through  in  the 
late  19th  century  (Carr,  2009). 

Around  the  turn  of  the  19th  century,  factories  had  to  purchase  customized  electric 
generation  equipment.  The  factory  had  to  install  the  equipment  and  train  personnel  on  the 
maintenance  of  the  equipment.  It  was  common  for  factories  to  hire  external  consultants  to 
perform  the  installation  and  training.  To  upgrade  the  generator  equipment,  or  change 
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vendors,  was  a  non-trivial  and  resource-intensive  task  requiring  preplanning.  All  the 
while,  the  vendors  of  the  electric  generation  hardware  made  a  large  profit  from  reselling 
the  same  technology  and  services  to  multiple  clients.  This  all  changed  when  the  idea  of 
selling  electricity  as  a  utility  was  implemented.  Economies  of  scale  were  obtained  by 
having  massive  electric  generating  locations  that  could  generate  enough  electricity  to 
power  many  homes  and  factories.  The  more  subscribers  there  were  to  the  utility  service, 
the  higher  the  utilization  rate  of  the  electric  generators,  which  in  turn  would  lower  the  per 
unit  utility  rate  for  consumers.  The  lower  rates  made  it  attractive  to  more  customers,  as  it 
now  was  more  cost  effective  to  purchase  electricity  from  a  utility  than  to  generate  it  in- 
house.  It  has  been  proposed  that  computing  is  following  a  similar  path  as  electricity  did 
and  that  in  the  not  too  distant  future  computing  will  be  offered  as  a  utility. 


Figure  4.  Power  Plant  circa  1904  from  (Leduc,  n.d.) 


At  the  turn  of  the  twenty-first  century,  Internet  usage  was  fundamentally 
changing,  in  large  part  due  to  advancements  in  software  development  and  the  widespread 
use  of  standards.  Software  development  was  transitioning  from  a  traditional  institutional 
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form  to  an  open  source  producer-subscriber  concept.  This  decentralized  approach 
allowed  for  applications  to  quickly  be  developed  with  Web  services  leading  to  the  rise  of 
social  media,  social  sites,  such  as  MySpace™,  FaceBook™,  YouTube™,  and  Twitter™, 
rapidly  became  household  names.  A  large  percentage  of  the  population  began  utilizing 
Web-based  social  networking  services  at  home  and  in  the  workplace,  and  became 
comfortable  with  storing  and  processing  data  at  a  location  other  than  their  own  computer. 
These  along  with  other  factors,  along  with  the  economic  downturn  in  2007-2010, 
culminated  in  today’s  intense  interest  in  cloud  computing. 

2.  Definition 

The  federal  CIO,  charged  with  leveraging  cloud  computing,  asked  the  National 
Institute  of  Standards  and  Technology  (NIST)  to  lead  federal  efforts  on  standards  for  data 
portability,  cloud  interoperability,  and  security  (“Summary  of  NIST  Cloud  Computing 
Standards  Development  Efforts,”  2010).  NIST  serves  as  the  government  lead,  working 
with  other  government  agencies,  industry,  academia,  Standards  Development 
Organizations  (SDO),  and  others  to  leverage  appropriate  existing  standards  and  to 
develop  cloud  computing  standards  where  gaps  exist  (Kundra,  2010).  Cloud  computing 
has  been  defined  by  NIST  as: 

a  model  for  enabling  convenient,  on-demand  network  access  to  a  shared 
pool  of  configurable  computing  resources  (e.g.,  networks,  servers,  storage, 
applications,  and  services)  that  can  be  rapidly  provisioned  and  released 
with  minimal  management  effort  or  service  provider  interaction” 
(“Summary  of  NIST  Cloud  Computing  Standards  Development  Efforts, 

2010). 

One  vision  of  the  future  of  cloud  computing  is  massively  scalable  data  centers 
that  fonn  a  seamless  infrastructure  capable  of  remotely  running  applications  and  storing 
data  that  can  be  accessed  from  any  connected  device  over  the  Internet. 

Clouds  offer  a  virtual  environment  for  hosting  user  applications  on  one  or  many 
servers  (physical  or  virtual)  making  clouds  particularly  compelling  for  applications  that 
have  unpredictable  usage  demands.  If  a  company  or  project  is  not  sure  if  they  will  need 
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five  or  fifty  servers  over  the  next  few  months,  provisioning  cloud  services  may  be  a 
viable  solution.  Or,  if  a  company  needs  to  quickly  ramp  up  resources  to  handle  a  short 
spike,  then  a  cloud  can  be  a  way  to  avoid  investing  in  hardware  that  could  be  grossly 
underutilized  for  the  majority  of  the  time. 


Figure  5.  Virtualization  from  (“Server  consolidation  and  virtualization,”  2010) 


Cloud  computing  can  exist  without  leveraging  virtualization  however;  users  will 
not  obtain  the  same  level  of  efficiency  or  Return  on  Investment  (ROI)  as  if  virtualization 
were  utilized.  Virtualization  was  invented  by  IBM  in  the  1960s  at  a  time  when  users  were 
using  key  punches  and  submitting  batch  jobs  based  on  a  server  time-sharing  system 
(Brodkin,  2009).  Virtualization,  through  the  use  of  hypervisors,  provided  a  mechanism 
for  allowing  multiple  users  to  utilize  the  same  mainframe  through  a  virtual  machine 
(VM)  without  risking  the  stability  of  the  entire  mainframe.  VMs  made  it  possible  for 
users  to  deploy  beta  version  software  for  testing  without  the  need  for  another  expensive 
mainframe.  Virtualization  is  made  possible  through  hypervisors  and  exokemels. 
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A  hypervisor  allows  multiple  operating  systems  (OS)  to  run  concurrently  on  a 
single  physical  piece  of  hardware  through  the  interception  of  inputs  and  outputs  (I/O)  and 
interrupt  calls  to  the  physical  hardware.  The  hypervisor  monitors  the  execution  of  the  OS 
for  these  instruction  calls  and  then  allocates  system  resources  (e.g.,  memory  address 
ranges,  CPU  cycles)  just  as  the  physical  hardware  would,  essentially  cloning  the  physical 
machine.  However,  since  multiple  OSs  are  running  on  the  same  physical  hardware,  the 
hypervisor  must  keep  tables  mapping  what  resources  each  OS  has  requested  and  is  using. 
That  way  the  hypervisor  does  not  hand  out  a  physical  resource  that  another  OS  is 
utilizing. 

Another  way  to  provide  virtualization  is  to  partition  the  physical  machines’ 
hardware  giving  each  user  a  subset  of  the  physical  resources.  This  partition  is  made 
possible  through  a  program,  running  in  kernel  mode,  called  an  exokemel.  The 
exokernaTs  primary  function  is  to  allocate  resources  to  users  and  only  allow  users  access 
to  the  resources  that  were  allocated  to  that  user.  The  advantage  of  an  exokernel  model  is 
that  it  saves  a  layer  of  mapping.  In  other  designs  each  VM  has  its  own  disk  with  blocks 
running  from  0  to  some  maximum.  The  VM  monitor  program  must  maintain  tables  to 
remap  disk  addresses  (and  all  other  resources)  whereas  the  exokernel  model  must  only 
keep  up  with  which  VM  has  been  assigned  what  resource  (Tanenbaum,  2008). 

VM  use  is  not  limited  to  servers.  VMs  have  multiple  uses  within  software 
development  such  as:  reducing  the  number  of  OSs  a  developer  must  maintain,  providing 
a  common  execution  environment,  and  providing  a  deployment  mechanism  for 
preinstalled  and  preconfigured  software.  If  a  software  package  will  be  deployed  to  four 
different  OSs,  then  prior  to  deployment  the  developer  should  test  the  software  package 
out  on  each  of  the  target  OSs.  VMs  allow  the  developer  to  keep  an  unlimited  number  of 
different  machine  configurations  stored  on  the  same  physical  hardware  removing  the 
necessity  for  keeping  multiple  physical  machines  or  hard  drives  simply  for  the  purpose  of 
testing.  VMs  can  also  be  used  on  a  smaller  scale  to  provide  a  common  execution 
environment  regardless  of  the  host  OS.  For  instance,  the  Java  programming  language 
uses  a  VM  called  the  Java  Virtual  Machine  (JVM)  as  a  means  of  ensuring  that  compiled 


16 


Java  bytecode  will  execute  properly  regardless  of  the  executing  host  OS.  JVM  is  the 
environment  where  the  Java  programs  execute  and  is  the  instance  of  the  Java  Runtime 
Environment  (JRE).  If  implemented  properly,  the  executing  Java  program  can  be  checked 
for  safety  and  executed  in  a  protected  environment,  the  JVM,  to  prevent  the  program 
from  doing  anything  unauthorized  or  damaging  the  host  OS  (Tanenbaum,  2008).  Just  as 
with  a  physical  machine,  it  is  the  software  that  makes  a  VM  useful.  When  a  VM  is  mixed 
with  software  you  get  a  virtual  appliance.  Deploying  a  preinstalled  and  preconfigured 
application  appliance  is  far  easier  than  preparing  a  system,  installing  the  application,  and 
configuring  and  setting  it  up.  A  virtual  appliance  is  not  a  VM,  but  rather  a  software  image 
containing  a  software  stack  designed  to  run  inside  a  VM.  (Shanna,  2008). 

From  a  server  operations  and  backup  and  restore  perspective,  virtualization  can  be 
a  tremendous  timesaver.  For  example,  the  initial  configuration  of  the  operating  system  for 
a  server,  along  with  the  software  to  run  on  that  server  can  take  hours,  if  not  days,  to 
configure.  With  virtualization,  that  initial  work  is  done  once,  and  the  resulting  standard 
image  is  saved  to  be  deployed  onto  physical  hardware  as  needed.  This  process  can  be 
done  in  as  little  as  a  few  seconds  to  minutes  and  repeated  as  often  as  needed. 

Regardless  of  how  it  is  accomplished,  virtualization  allows  multiple  users  to  share 
hardware  and  software  services  without  knowing  and  preferably  without  interfering  with 
each  other’s  processes.  Virtualization  enables  a  provider  to  present  an  environment  to 
clients  that  appears  to  be  completely  their  own  enviromnent  while  still  gaining  the 
economy  of  scale  that  multiple  tenants  provide,  (“Summary  of  NIST  Cloud  Computing 
Standards  Development  Efforts,”  2010)  and  reducing  facility  requirements,  such  as 
power  and  cooling  (Carter  &  Rajamani,  2010). 

3.  Essential  Characteristics 

The  five  recognized  essential  characteristics  of  a  cloud  computing  environment 
are  shown  in  Figure  6: 
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Rapid  elasticity 


Broad  network  access 


On-demand  self  service 


Figure  6.  Essential  Cloud  Computing  Characteristics  from  (“Federal  Guidance 
Needed  to  Address  Control  Issues  with  Implementing  Cloud  Computing,” 

2010) 


a.  Rapid  Elasticity 

Rapid  elasticity  is  defined  as  the  ability  to  scale  resources  both  up  and 
down  as  needed.  Resources  that  can  be  scaled  up  or  down  include  the  number  of 
processors,  network  bandwidth,  storage  space,  or  software  instances  needed  by  a  client. 
This  scaling  can  occur  within  a  matter  of  minutes  or  hours  instead  of  weeks  or  months. 
Cloud  computing  clients  do  not  have  to  evaluate  server  cage,  floor  space,  power,  or 
cooling  requirements  before  provisioning  a  new  server.  All  they  must  do  is  provide 
payment  for  the  capability  that  is  required  (“Summary  of  NIST  Cloud  Computing 
Standards  Development  Efforts,”  2010).  To  the  client,  the  cloud  appears  to  be  infinite, 
and  the  client  can  provision  as  many  units  of  cloud  services  as  needed  (Jackson,  n.d.). 

b.  Measured  Service 

Measured  service  refers  to  the  cloud  provider  monitoring  and  controlling 

all  aspects  of  the  cloud  service.  Cloud  systems  automatically  control  and  optimize 

resource  use  by  leveraging  a  metering  capability  at  some  level  of  abstraction  appropriate 

to  the  type  of  service  (e.g.,  storage,  processing,  bandwidth,  and  active  user  account). 
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Resource  usage  can  be  monitored,  controlled,  and  reported  providing  transparency  for 
both  the  provider  and  consumer  of  the  utilized  service  (“Summary  of  NIST  Cloud 
Computing  Standards  Development  Efforts,”  2010).  Tracking  resource  usage  is  crucial 
for  billing,  access  control,  resource  optimization,  capacity  planning  and  other  tasks 
(Jackson,  n.d.). 


c.  On-Demand  Self  Service 

The  on-demand  and  self-service  aspect  means  that  a  consumer  can 
unilaterally  provision  computing  capabilities,  such  as  server  time  and  network  storage,  as 
needed  automatically  without  requiring  human  interaction  with  each  service’s  provider 
(“Summary  of  NIST  Cloud  Computing  Standards  Development  Efforts,”  2010). 

d.  Broad  Network  Access 

Broad  network  access  means  that  the  cloud  provider’s  capabilities  are 
available  over  the  network  and  can  be  accessed  through  standard  mechanisms  that 
promote  use  by  heterogeneous  thin-client  or  thick-client  platforms  (e.g.,  mobile  phones, 
laptops,  and  PDAs)  (“Summary  of  NIST  Cloud  Computing  Standards  Development 
Efforts,”  2010). 


e.  Resource  Pooling 

With  Resource  Pooling,  the  provider’s  computing  resources  are  pooled  to 
serve  multiple  consumers  using  a  multi-tenant  model,  with  different  physical  and  virtual 
resources  dynamically  assigned  and  reassigned  according  to  user  demand.  There  is  a 
sense  of  location  independence  in  that  the  client  generally  has  no  control  or  knowledge 
over  the  exact  location  of  the  provided  resources  but  may  be  able  to  specify  location  at  a 
higher  level  of  abstraction  (e.g.,  country,  state,  or  datacenter).  However,  this  may  change 
especially  given  the  different  treatment  of  security  and  privacy  from  one  nation  to 
another,  and  the  advent  of  cloud  computing  used  for  processing  and  storing  data  that  is 
sensitive  to  national  security.  Examples  of  resources  include  storage,  processing, 
memory,  network  bandwidth,  and  virtual  machines.  Utilization  by  multiple  tenants  is  part 
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of  what  makes  the  “as  a  service”  portion  of  cloud  computing  an  attractive  business 
proposition.  Without  multiple  clients  to  use  a  service,  the  cost  of  maintaining  that  service, 
and  subsequently  the  cost  for  a  client  to  subscribe  to  that  service,  could  be  so  large  that  a 
service  provider  could  not  make  sufficient  returns  on  investment.  The  ability  to  have 
multiple  clients  using  the  same  platform  permits  economies  of  scale  to  come  into  play. 
The  more  clients  are  able  to  use  the  same  platfonn  will  lower  the  overhead  costs  that  are 
passed  on  to  each  client  (“Summary  of  NIST  Cloud  Computing  Standards  Development 
Efforts,”  2010). 

4.  Architectures 

All  of  the  terms  below  share  common  characteristics,  such  as:  a  purchasing  model 
of  pay-as-you-go  or  pay-as-you-use-it,  on  demand  scalability  of  the  amount  you  use,  the 
concept  that  there  will  be  many  people  or  multiple  tenants  using  the  service 
simultaneously,  and  virtualization  (Abbott  &  Fisher,  2010). 

The  three  recognized  cloud  architectures  are: 

a.  Infrastructure  as  a  Service  (IaaS) 

The  term  Infrastructure  as  a  Service  (IaaS)  refers  to  offerings  of 
fundamental  computer  resource  infrastructure  such  as  processing  power,  servers,  storage, 
networking  components  and  bandwidth  as  a  service.  This  method  is  typically  a  pay-as- 
you-use  model  for  what  previously  required  either  capital  expenditure  to  purchase 
outright,  long-term  leases,  or  month-to-month  subscriptions  for  partial  tenancy  of 
physical  hardware.  The  capability  provided  to  the  consumer  is  the  ability  to  provision 
these  core  computing  resources  such  that  the  consumer  is  able  to  deploy  and  run  arbitrary 
software,  which  can  include  operating  systems  and  applications.  The  consumer  does  not 
manage  or  control  the  underlying  cloud  infrastructure  but  has  control  over  operating 
systems;  storage,  deployed  applications,  and  possibly  limited  control  of  select  networking 
components  (e.g.,  host  firewalls,  load  balancers)  (“Summary  of  NIST  Cloud  Computing 
Standards  Development  Efforts,”  2010). 
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b.  Platform  as  a  Service  (PaaS) 

The  term  Platform  as  a  Service  (PaaS)  refers  to  offerings  of  a  computing 
platform  and/or  solution  stack  as  a  service.  The  platform  typically  is  an  application 
framework  that  typically  consumes  cloud  infrastructure  and  supporting  cloud 
applications.  The  capability  provided  to  the  consumer  is  to  deploy  onto  the  cloud 
infrastructure  consumer-created  or  acquired  applications  created  using  programming 
languages  and  tools  supported  by  the  provider.  PaaS  facilitates  deployment  of 
applications  without  the  cost  and  complexity  of  buying  and  managing  the  underlying 
hardware  and  software  layers  (Jackson,  n.d.).  The  consumer  does  not  manage  or  control 
the  underlying  cloud  infrastructure,  including  network,  servers,  operating  systems, 
storage,  but  the  consumer  has  control  over  the  deployed  applications  and  possibly 
application  hosting  environment  configurations  (“Summary  of  NIST  Cloud  Computing 
Standards  Development  Efforts,”  2010). 

c.  Software  as  a  Service  (SaaS) 

The  capability  provided  to  the  consumer  is  to  use  the  provider’s 
applications  running  on  a  cloud  infrastructure.  The  applications  are  accessible  from  client 
devices  through  a  thin-client  interface  such  as  a  Web  browser  (e.g.,  Web-based  e-mail). 
The  consumer  uses  an  application  but  does  not  manage  or  control  the  underlying  cloud 
infrastructure,  including  network,  servers,  operating  systems,  storage,  or  even  individual 
application  capabilities  on  which  it  is  running,  with  the  possible  exception  of  limited 
user-specific  application  configuration  settings  (“Summary  of  NIST  Cloud  Computing 
Standards  Development  Efforts,”  2010). 

5.  Deployment  Models 

Within  clouds  there  is  not  a  “one  size  fits  all”  deployment.  Different  companies 
and  sectors  have  different  requirements  and  concerns.  NIST  has  theorized  that  there  are 
four  different  ways  to  deploy  a  cloud.  Each  deployment  model  has  its  own 
characteristics,  typically  with  varying  security  requirements. 
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The  four  recognized  cloud  deployment  models  are: 


a.  Public  Cloud 


Public  Cloud 


Figure  7.  Public  Cloud  from  (“Cloud  Computing  Use  Cases  White  Paper  v3.0,” 

2010) 

The  cloud  infrastructure  is  made  available  to  the  general  public  or  a  large 
industry  group  and  is  owned  by  an  organization  selling  cloud  services  (“Summary  of 
NIST  Cloud  Computing  Standards  Development  Efforts,”  2010).  In  simple  terms,  public 
cloud  services  are  characterized  as  being  available  to  clients  from  a  third-party  service 
provider  via  the  Internet.  The  term  “public”  does  not  always  mean  free,  even  though  it 
can  be  free  or  fairly  inexpensive  to  use.  A  public  cloud  does  not  mean  that  a  user’s  data  is 
publically  visible;  providers  typically  provide  an  access  control  mechanism  for  their 
users.  Public  clouds  provide  an  elastic,  cost  effective  means  to  deploy  solutions  (Jackson, 
n.d.).  An  external  or  public  cloud  is  provided  by  an  external  independent  entity,  typically 
a  cloud  service  provider. 
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b. 


Private  Cloud 


Msssam 

Figure  8.  Private  Cloud  from  (“Cloud  Computing  Use  Cases  White  Paper  v3.0,” 

2010) 

The  cloud  infrastructure  is  operated  solely  for  an  organization.  It  may  be 
managed  by  the  organization  or  a  third  party  and  may  exist  on  or  off  premise  (“Summary 
of  NIST  Cloud  Computing  Standards  Development  Efforts,”  2010).  A  private  cloud  is  an 
internal  deployment  where  cloud  computing  capabilities  are  planned,  architected, 
acquired,  and  implemented  to  support  internal  business  requirements  (Jackson,  n.d.),  all 
while  mitigating  risks  associated  with  security,  privacy,  legal  requirements,  and  the 
relative  immaturity  of  the  cloud  industry  and  associated  technology.  Private  clouds  offer 
many  of  the  benefits  of  a  public  cloud  computing  environment,  such  as  being  elastic  and 
service-based,  but  data  and  processes  are  managed  within  the  organization.  Private  clouds 
offer  the  organization  and  user  greater  control  of  the  cloud  infrastructure,  and  can 
improve  security  and  flexibility,  because  user  access  and  the  networks  used  are  controlled 
by  the  organization.  However,  it  is  important  to  note  that  even  within  Private  clouds 
communications  still  travel  over  the  Internet. 
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c. 


Community  Cloud 


Community 


Cloud 

(Public  or  Private) 


Compute 

Services 


Figure  9.  Community  Cloud  from  (“Cloud  Computing  Use  Cases  White  Paper 

v3.0,”  2010) 


The  cloud  infrastructure  is  shared  by  several  organizations  and  supports  a 
specific  community  that  has  shared  concerns  or  interests,  such  as  specific  security 
requirements,  a  common  mission,  policy,  or  compliancy  issues.  It  may  be  managed  by 
the  organizations  or  a  third  party  and  may  exist  on  or  off  premise  (“Summary  of  NIST 
Cloud  Computing  Standards  Development  Efforts,”  2010).  The  members  of  the 
community  share  access  to  the  data  and  applications  in  the  cloud.  In  essence,  it  is  a  semi¬ 
private  cloud  formed  to  meet  the  needs  of  a  set  of  related  stakeholders  that  have  common 
requirements  or  interests.  It  may  be  private  for  its  stakeholders,  or  may  be  a  hybrid  that 
integrates  the  respective  private  clouds  of  the  members,  yet  enables  the  sharing  and 
collaboration  across  their  individual  clouds  by  exposing  data  or  resources  into  the 
community  cloud  (Jackson,  n.d.). 
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d.  Hybrid  Cloud 


Public  Cloud 


Public  Cloud 


HilBH 

Figure  10.  Hybrid  Cloud  from  (“Cloud  Computing  Use  Cases  White  Paper  v3.0,” 

2010) 


The  cloud  infrastructure  is  a  composition  of  two  or  more  clouds  (private, 
community,  or  public)  that  remain  unique  entities  but  are  bound  together  by  standardized 
or  proprietary  technology  that  enables  data  and  application  portability  (e.g.,  cloud 
bursting)  (“Summary  of  NIST  Cloud  Computing  Standards  Development  Efforts,”  2010). 

Rather  than  throw  out  local  applications  and  use  the  cloud  exclusively,  or, 
conversely,  rely  on  local  applications  only  and  ignore  the  cloud,  the 
prevailing  wisdom  is  to  use  a  combination  of  local  applications  and  the 
cloud. 


(O'Neill,  2009) 

Hybrid  clouds  typically  blend  a  combination  of  internal  cloud  and  external 
cloud-enabled  resources  that  allows  organizations  to  take  advantage  of  the  cost 
economics  of  external  third-party  clouds  while  mitigating  some  of  the  risks  by 
maintaining  an  internal  private  cloud  for  critical  processes  and  data.  This  hybrid  approach 
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lends  itself  to  an  incremental  deployment  allowing  businesses  to  test  out  the  cloud  while 
not  risking  everything  on  the  success  or  failure  of  a  specific  vendor. 

6.  Government  Cloud  Efforts 

The  following  sub-sections  provide  recent  examples  of  how  federal  agencies  are 
using  cloud  computing  technologies.  The  examples  presented  below  provide  a  small 
sampling  of  how  cloud  computing  technologies  are  currently  being  used  within  the 
federal  government — such  as  establishing  a  private  cloud,  standardizing  the  procurement, 
accreditation,  and  certification  of  cloud  computing  technologies,  establishing  a  massive 
cloud-to-support  research  and  moving  non-critical  processes  to  a  cloud-based 
environment. 

a.  Defense  Information  Systems  Agency  (DISA)  Rapid  Access 
Computing  Environment  (RACE) 

The  Defense  Infonnation  Systems  Agency  (DISA)  provides  Infonnation 
Technology  support  to  the  Department  of  Defense  (DoD).  In  2008,  DISA  began 
leveraging  cloud  computing  by  creating  its  own  secure  private  cloud,  the  Rapid  Access 
Computing  Environment  (RACE).  RACE  provides  on-demand  server  space  for 
development  teams  through  the  use  of  virtual  server  technology.  By  using  virtualization 
technologies,  DISA  has  reduced  the  number  of  physical  servers  required  to  support  DoD 
missions,  and  for  the  physical  servers  that  remain  DISA  shares  the  operating  costs 
amongst  the  users  of  the  virtual  servers.  Users  pay  a  fee  for  provisioning  units  of  cloud 
services,  which  is  then  used  by  DISA  to  pay  for  the  physical  hardware  and  associated 
software  that  houses  all  of  the  virtual  servers  and  subsequent  required  resources  (e.g., 
energy,  software,  human  resources).  Within  this  virtual  environment,  users  can  use  a  self- 
service  portal  to  provision  computing  resources  in  50  GB  increments  with  the  guarantee 
that  the  environment  meets  DoD  Certification  and  Accreditation  (C&A)  standards.  The 
capability  that  RACE  offers  to  the  DoD  community  has  reduced  the  server  provisioning 
process  from  being  at  best  case  a  multiple-week  process  to  a  mere  twenty-four  hour 
process.  According  to  DISA,  personnel  can  expect  the  same  level  of  service  and 
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availability  when  using  RACE  over  a  traditional  environment  (Kundra,  2010).  However, 
if  users  discover  that  RACE  simply  will  not  meet  their  needs,  DISA  has  processes  in 
place  for  exiting  the  RACE  cloud  environment.  DISA  has  established  a  strict  data 
cleansing  process  for  removing  an  application  completely  from  the  RACE  platform. 
Since  the  inception  of  this  cloud-based  solution,  hundreds  of  military  applications 
including  command  and  control  systems,  convoy  control  systems,  and  satellite  programs 
have  been  developed  or  tested  on  RACE  (“Rapid  Access  Computing  Environment 
(RACE),”  2010). 

b.  Federal  Risk  and  Authorization  Management  Program 
(FedRAMP) 

The  Federal  Risk  and  Authorization  Management  Program  (FedRAMP)  is 
a  government-wide  risk  management  program,  formed  by  the  Cloud  Computing 
Advisory  Council,  focused  on  large  outsourced  and  multi-agency  systems.  FedRAMP 
will  provide  joint  authorizations  and  continuous  security  monitoring  of  shared  IT  services 
for  federal  departments  and  agencies  that  enter  contracts  with  outside  providers.  It  seeks 
to  create  a  uniform  set  of  Federal  Infonnation  Security  Management  Act  (FISMA) 
compliant  security  standards  available  for  leverage  government-wide  when  procuring, 
certifying,  and  accrediting  cloud  computing  offerings.  Initially,  the  program  will  focus  on 
cloud  computing  but  will  expand  to  other  domains  as  the  program  matures  (“Federal  Risk 
and  Authorization  Management  Program  (FedRAMP),”  2010).  FedRAMP  will  help  lead 
to  the  development  of  common  security  requirements  for  specific  types  of  systems, 
provide  ongoing  risk  assessments,  encourage  better  system  integration,  and  drastically 
reduce  duplication  of  effort  and  associated  costs.  But,  perhaps  most  of  all,  it  has  the 
potential  to  bring  a  common-sense,  approve-once,  use-often  approach  that  has  long 
eluded  government  IT  acquisitions  (Kash,  2010). 

These  cloud  security  standards  will  allow  agencies  to  expedite  the 
acquisition  of  cloud  products  through  the  use  of  collaboratively  developed  security  and 
accreditation  baselines.  The  ultimate  goal  being  to  streamline  the  duplicative  process  of 
certifying  the  security  of  applications  destined  to  be  shared  by  multiple  governmental 
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agencies.  Under  the  current  process,  vendors  must  certify  products  with  every  agency 
even  though  each  agency  may  have  identical,  or  nearly  identical,  security  requirements. 
This  is  a  highly  inefficient  C&A  model  that  leads  to  a  duplication  of  effort  both  within 
agencies  and  the  vendors.  If  FedRAMP  works  as  intended,  then  cloud  vendors  could 
deliver  to  a  single  set  of  baseline  standards,  which  will  encourage  innovation  and  bolster 
competition  (Oltsik,  2010).  On  the  government  side,  FedRAMP’s  efforts  would  allow 
agencies  to  reduce  security  compliance  expenditures,  expedite  acquisition  times,  and 
provide  consistent  integration  with  government-wide  security  efforts,  without  sacrificing 
any  unique  security  needs  of  their  agency  (Chabrow,  2010). 

c.  National  Aeronautics  and  Space  Administration  (NASA)  Nebula 

Nebula,  NASA’s  open-source  cloud-computing  platfonn,  was  developed 
to  provide  an  easily  quantifiable  and  improved  alternative  to  building  additional 
expensive  data  centers  and  to  provide  an  easier  way  for  NASA  scientists  to  share  large, 
complex  data  sets  with  external  partners  and  the  public.  It  consists  of  hardware  and 
software  housed  in  shipping  containers  at  NASA’s  Ames  Research  Center.  Each  shipping 
container  data  center  supports  100  times  the  file  size  and  ten  times  the  networking  speed 
of  the  most  powerful  commercial  cloud  environment  and  can  provide  up  to  fifteen 
petabytes  (one  petabyte  equals  one  million  gigabytes)  of  storage  or  15,000  central 
processing  units  (CPU)  (Hoover,  2010a).  Nebula  allows  NASA  to  process,  store  and 
upload  thousands  of  high-resolution  images  and  over  100  terabytes  of  data  and  has 
assisted  NASA  with  the  processing  of  high  resolution  images  of  the  Moon  and  Mars  for 
use  in  worldwide  mapping.  During  this  effort,  data  from  satellites  was  processed  and  then 
sent  to  Microsoft  for  placement  on  a  three-dimensional  map  of  the  world.  In  a  traditional 
IT  environment,  new  infrastructure  would  have  to  be  procured  in  order  to  have  enough 
computing  and  storage  resources  to  handle  the  massive  amounts  of  data.  The 
procurement  process  alone  would  have  taken  months,  as  engineers  would  first  need  to 
justify  why  the  project  requires  new/additional  hardware/software,  retrieve  and  supply 
multiple  quotes,  wait  on  an  approval  decision,  wait  on  the  hardware/software  to  be 
ordered  and  delivered,  and  then  finally  the  system  administrators  could  begin  the  job  of 
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configuring  the  new  hardware/software  for  use  by  the  engineers.  At  the  conclusion  of  the 
project,  NASA  would  be  left  with  a  pile  of  legacy  hardware  that  may  or  may  not  be  able 
to  be  utilized  for  future  projects.  Through  the  utilization  of  Nebula,  NASA  was  able  to 
provision  the  virtual  machines  and  have  them  up  and  running  in  a  manner  of  hours.  This 
ability  to  quickly  ramp  up  and  down  allows  the  engineers  to  focus  on  mission-critical 
activities  instead  of  IT  infrastructure  (West,  2010).  Through  the  hosting  of  the  federal 
government’s  USASpending.gov  Web  site,  Nebula  has  demonstrated  how  a  cloud¬ 
computing  platform  hosted  by  one  agency  can  be  utilized  by  another  federal  entity 
(“NASA  Flagship  Initiatives:  Nebula,”  2010). 


Figure  1 1 .  NASA  Nebula  Container  from  (“NASA  Flagship  Initiatives:  Nebula,” 

2010) 


In  addition  to  expediting  the  process  of  standing  up  resources  for  new 
efforts,  Nebula  also  assists  NASA  with  addressing  a  side  effect  of  having  decade-long 
missions,  keeping  servers  and  data  acquisition  systems  under  configuration  control.  Since 
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NASA  space  exploration  missions  can  take  over  ten  years  to  develop,  the  resources 
needed  to  process  the  data  coming  back  are  usually  scheduled  and  procured  well  before 
launch  and  placed  in  a  configuration  lock  down.  Missions,  however,  can  be  delayed  at  a 
late  stage,  cancelled  altogether,  or  last  much  longer  than  originally  anticipated  leading  to 
IT  infrastructure,  which  is  not  needed  or  is  required,  to  be  in  configuration  control  for 
longer  than  expected.  Nebula's  cloud  services  allow  NASA  to  be  much  more  flexible  and 
responsive  to  actual  mission  needs,  scaling  resources  up  or  down  as  the  actual 
requirements  of  the  mission  develop. 

Currently,  Nebula  is  an  IaaS  implementation  allowing  IaaS  customers  to 
unilaterally  provision,  manage,  and  decommission  computing  capabilities  on  an  as- 
needed  basis  through  a  Web  interface  or  set  of  command-line  tools.  NASA  plans  to  begin 
offering  a  PaaS  in  late  2010  and  a  Database  as  a  Service  (DaaS)  in  201 1. 

d.  Department  of  Energy  (DOE)  Cloud  Computing  Migration 

The  Department  of  Energy  is  exploring  cost  and  energy  efficiencies  that 
can  result  from  leveraging  cloud  computing.  The  Lawrence  Berkeley  National  Labs 
(LBL)  initiative  is  exploring  how  to  use  cloud  computing  to  address  needs  across  the 
enterprise,  in  specific  business  services  and  in  scientific  study.  According  to  the  DOE 
CIO:  has  deployed  over  2,300  mailboxes  on  Google  Federal  Premier  Apps,  and 

plans  to  have  5,000  e-mail  accounts  deployed  by  August  2010.  This  solution  uses  a  LBL 
Identity  Management  System  to  provide  authentication” .  LBL  small  and  medium  sized 
scientific  research  teams  are  also  utilizing  Google  Docs  and  Google  Sites  to  foster 
collaboration  and  community  documentation.  LBL  estimates  the  deployments  that  have 
already  been  made  will  save  $1.5  million  over  the  next  five  years  in  hardware,  software 
and  labor  (Kundra,  2010). 

e.  Department  of  Interior  (DOI)  Agency-wide  E-mail 

The  Department  of  the  Interior  is  pursuing  a  SaaS  cloud  computing  model 
for  e-mail.  DOI  has  80,000  e-mail  users  who  are  widely  dispersed  across  the  United 
States  and  are  currently  supported  by  a  complex  messaging  infrastructure  comprised  of 
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more  than  a  dozen  different  e-mail  systems.  “By  implementing  e-mail  using  an  external 
commercial  SaaS  model,  the  department  expects  to  provide  improved  service  to  its 
80,000  users  for  one-third  the  amount  of  money  that  it  spends  today"  (Kundra,  2010). 
The  department  is  moving  forward  with  this  project  with  an  anticipated  completion  date 
in  Fiscal  Year  2011. 

7.  Cloud  Concerns 


Figure  12.  Stonny  Road  Ahead  from  (Price,  2007) 


Although  cloud  computing  promises  to  provide  enterprises  and  IT  with  relief 
from  numerous  pain  points,  utilizing  cloud  deployment  methods,  either  individually  or  in 
combination,  significantly  alters  an  organization’s  risk  profile  (Christiansen  et  ah,  2010). 
Moving  to  a  cloud-based  environment  may  not  be  appropriate  in  all  circumstances,  and 
whether  it  is  a  viable  option  will  depend  on  how  it  is  used  and  how  many  risks  can  be 
mitigated.  Just  as  with  any  other  new  technology  or  service,  moving  to  a  cloud 
environment  should  be  done  in  a  careful  balanced  way  and  only  after  thorough  planning, 
research,  risk  assessment,  and  pathfinder  pilots  have  been  performed.  Business  goals 
should  be  defined,  a  risk-benefit  analysis  should  be  performed  to  determine  whether 
moving  to  a  cloud  is  applicable,  worst-case  scenarios  should  be  examined  to  identify 
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what  processes  are  jeopardized,  and  contingency  plans  should  be  created  for  each 
scenario,  e.g.,  scenarios  such  as  what  happens  if  there  is  a  security  breach  or  what 
happens  if  the  cloud  is  unreachable  for  various  lengths  of  time.  The  section  below  lists 
some  of  the  many  areas  of  concern  that  potential  cloud  adopters  should  investigate  prior 
to  moving  to  the  cloud. 

a.  Security 

First  instincts  when  outsourcing  control  and  processing  of  data  is  to  be 
concerned  that  doing  so  will  lower  an  organization’s  overall  security  posture;  however, 
that  may  not  always  be  the  case.  Cloud  computing  has  the  potential  to  enhance  security 
while  also  introducing  additional  vulnerabilities.  In  most  organizations,  security  is  either 
understaffed  or  is  another  duty  as  assigned  and  not  the  primary  competency  of  the  IT 
staff.  Additionally  in  the  past,  keeping  data  secure  was  much  less  challenging  than  it  is 
today.  File  formats  were  likely  proprietary,  data  was  placed  in  silos,  relationships 
between  data  producers  and  consumers  were  closely  coupled,  and  fewer  people 
worldwide  had  access  to  technology.  Today  organizations  could  potentially  obtain  better 
security  by  moving  to  a  cloud  environment  where  the  provider  has  dedicated  security 
personnel  whose  sole  mission  is  to  secure  the  cloud  infrastructure  (Robert  Mullins,  2010). 
From  a  patching  standpoint,  cloud  computing  holds  the  potential  to  drastically  change  the 
losing  game  of  continual  patching  and  IT  device  remediation  through  the  distribution  of 
standardized  locked  down  machine  images  (Gourley,  2009).  Through  the  use  of 
virtualization  and  automation,  cloud  computing  will  expedite  the  implementation  and 
deployment  of  secure  configurations  for  VM  images.  When  vulnerabilities  are  detected, 
they  could  be  managed  more  rapidly  and  uniformly. 

Conversely,  having  a  private  DoD  cloud  could  also  lead  to  increased 
coordinated  attacks  on  the  cloud.  Due  to  the  high  collection  of  data  within  a  cloud,  a  DoD 
private  cloud  would  create  an  extremely  high-profile  target  for  hackers,  foreign  states, 
and  terrorist  organizations.  When  you  put  more  eggs  in  one  basket,  the  prize  for  breaking 
into  the  basket  becomes  much  larger.  If  a  DoD  cloud  were  successfully  breached,  then 
the  attacker  could  move  laterally  and  capture  more  information  than  if  a  single 
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installation  were  breached.  From  a  national  security  perspective  the  amount  of  damage 
that  could  be  inflicted  on  the  DoD  via  spyware,  botnets,  or  other  malicious  programs  is 
magnitudes  greater  than  in  non-cloud  environments.  Thus  it  is  important  that  all  security 
policies,  processes,  procedures,  and  controls  in  place  have  been  implemented  properly. 
From  an  adopter’s  standpoint,  it  is  important  to  understand  what  these  security  policies 
are,  what  the  incident  response  is  in  case  of  a  breech,  what  type  of  physical  security  is  in 
place,  what  data  protection  measures  are  in  place  (such  as  data  encryption  at  rest  and  in¬ 
motion),  what  type  of  logging  and  auditing  are  employed,  how  information  leakage  is 
prevented,  how  data  is  segregated  in  multitenant  situations,  how  data  is  destroyed  after 
removal  from  the  cloud  (e.g.,  backups),  how  root  user  access  is  controlled,  who  all  has 
access  to  the  data,  and  so  on. 

Due  to  the  fluid  nature  of  cloud  computing,  it  is  difficult  to  even  detennine 
what  questions  we  should  be  asking  to  improve  the  security  and  privacy  that  clouds  can 
afford.  However,  a  few  of  the  many  questions  and  issues  related  to  providing  security 
assurances  within  a  cloud  environment  follow.  (Dinolt  &  Michael,  2010)  Fundamental 
questions  should  be  addressed  such  as  whether  current  architectures  are  adequate  for 
building  trusted  clouds,  or  if  new  architectures  are  needed.  Whether  current  OS  and 
application  security  approaches  will  scale  properly  to  a  cloud  computing  environment,  or 
if  new  approaches  will  need  to  be  taken  to  provide  a  trusted  OS.  Efforts  are  currently 
underway  within  the  Naval  Postgraduate  School  (NPS)  to  investigate  the  security 
policies,  models,  and  appropriate  architectures  to  address  the  OS  and  architecture 
questions.  More  detailed  information  regarding  NPS’s  research  can  be  found  in  the  DoD 
IAnewsletter  article  titled  “Establishing  Trust  in  Cloud  Computing”.  (Dinolt  &  Michael, 
2010) 

Since  users  and  developers  are  still  discovering  new  ways  to  apply  cloud 
technologies,  generating  new  expectations  surrounding  security  and  privacy,  it  will  be 
difficult  for  organizations  to  assess  and  manage  risks.  The  cloud  and  security 
communities  should  establish  a  common  set  of  ‘best  practices’  for  cloud  based  security 
policies/models,  addressing  items  such  as:  data  integrity,  protection  of  personally 
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identifiable  information,  data  destruction,  communications  security,  privacy, 
authentication,  and  so  on.  Several  vendors  have  formed  the  Cloud  Security  Alliance 
(CSA),  which  is  promoting  independent  research  into  best  practices  for  cloud  computing 
security.  These  communities  should  also  address  how  to  provide  checks  and  balances  to 
ensure  that  security  resources  are  correctly  implemented  and  maintained  in  the  cloud.  The 
best  practices  mentioned  above  could  be  used  by  customers  in  evaluating  the  offerings  of 
various  vendors.  Providers  must  find  the  fine  line  between  supplying  customers  with 
enough  transparency  into  security  protections  and  procedures  to  alleviate  concerns,  while 
not  providing  so  much  transparency  as  to  assist  malefactors.  This  transparency  begins 
with  the  service-level  agreement  (SLA). 

b.  Service  Level  Agreement  (SLA) 

An  SLA  is  part  of  a  service  contract  where  the  level  of  service  is  formally 
defined  between  a  provider  and  a  customer.  SLAs  are  common  in  network  services  and 
typically  measure  the  parameter  set  known  as  quality  of  service  (QoS).  It  is  important  to 
note  that  while  SLAs  are  important  they  cannot  guarantee  acceptable  performance,  SLAs 
can  only  punish  failure  to  perform  as  promised.  The  more  detailed  an  SLA  is  the  better. 
Typical  items  specified  within  an  SLA  include  pricing,  processes  and  procedures, 
business  continuity  requirements,  reliability,  data  safety,  security,  availability, 
maintainability,  performance  benchmarks,  provisioning  turnaround  times,  geographic 
restrictions,  encryption  requirements,  system  uptime,  resiliency,  responsiveness, 
emergency  contacts,  clauses  that  spell  out  the  rights  and  responsibilities  of  the  provider 
and  customer,  and  last  but  not  least,  violation  recourse  (Milbum,  2010). 

However,  current  commercial  cloud  implementations  do  not  offer 
adequate  SLAs,  their  control  tools,  or  machine-actionable  SLAs  yet.  The  SLAs  do  not 
offer  safety  audit  processes  or  regulations  for  storage  and  backup  of  data  that  is  managed 
in  the  cloud  (Cueli,  2010).  Another  stumbling  block  for  cloud  computing  SLAs  is  that 
enforcement  will  be  difficult  because  of  the  shared  nature  of  cloud  computing  (Avoyan, 
2010).  If  QoS  issues  arise,  it  will  be  extremely  difficult  to  determine  the  root  cause  for 
service  interruptions  due  to  the  complex  nature  of  the  shared  resources  (e.g., 

34 


infrastructure,  platfonn,  software)  each  of  which  are  integral  to  the  end  users’  experience. 
In  a  cloud  environment,  it  may  be  difficult  to  determine  who  should  be  held  accountable 
for  the  service  interruption. 

c.  Standards  and  Data  Portability 

Cloud  adopters,  both  public  and  private,  must  be  able  to  easily  store, 
access,  and  process  data  across  multiple  clouds;  weave  together  a  mesh  of  different 
services  to  meet  their  needs;  and  have  a  way  to  collaborate  with  business  partners  around 
the  globe.  However,  without  security,  management,  data  federation,  and  multi-tenancy 
standards  cloud  interoperability  will  not  be  fully  realized  and  adopters  will  experience 
vendor  lock-in.  Currently,  there  are  several  ongoing  efforts  within  focus  groups, 
nonprofits,  and  standards  bodies  to  create  community-recognized  standards  for  cloud 
computing.  Groups,  such  as  NIST,  CSA,  and  the  European  Network  and  Information 
Security  Agency,  are  all  working  on  cloud  computing  standards.  Without  these  standards, 
there  will  also  likely  be  Application  Prograimning  Interface  (API)  and  platform  lock-in. 
Without  formal  standards,  de  facto  standards  will  arise  from  vendor  and  customer 
interactions;  for  instance,  software  giants  Microsoft  and  Oracle  have  indicated  that  they 
will  press  on  with  developing  cloud  solutions  regardless  of  lock-in  and  let  standards  catch 
up  down  the  road,  if  they  can  (Clarke,  2010). 

Before  moving  to  a  cloud  environment,  organizations  should  have  a  well- 
documented  entrance  and  exit  strategy  for  critical  data  and  business  processes.  Distinct 
strategies  should  exist  for  importing  and  exporting  data  into  vendor-specific  software, 
transfonning  data  formats  to  and  from  formats  supported  by  the  vendor,  and  for 
transitioning  business  processes  from  a  traditional  environment  to  and  from  a  potentially 
proprietary-cloud  based  format  (DiMaio,  2009).  Any  organization  that  moves  data,  and 
more  importantly  business  processes  perfonned  in  embedded  software  services,  into  a 
cloud-based  environment  should  develop  an  exit  strategy.  If  the  move  to  the  cloud  does 
not  work  as  expected,  organizations  should  have  a  well-defined  process  for  retrieving 
both  their  data  and  business  processes  out  of  the  cloud  and  returning  it  to  a  traditional 
environment  (Prigge,  2010).  This  involves  knowing  what  the  cloud  provider  requires  to 
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remove  data  from  its  cloud  (such  as  RACE’S  strict  data  cleansing  process),  whether  the 
provider  has  any  utilities  to  assist  with  removing  the  data,  whether  the  data  will  need  to 
be  transformed  from  one  fonnat  to  another,  and  whether  the  provider  will  charge  for 
assisting  with  the  removal  of  the  data  from  the  cloud. 

8.  Steps  to  Cloud  Nirvana 

Moving  to  a  cloud  environment  is  not  merely  about  obtaining  cost  savings  and 
efficiencies  of  scale  with  hardware  and  software,  the  low-hanging  fruit.  The  move  is  also 
about  removing  unnecessary  constraints  tying  users  to  specific  hardware,  software,  and 
workflows  leading  to  increased  user  productivity.  Cloud  computing  has  the  potential  to 
change  computing  from  being  application-centric  to  a  data-centric  view  of  information 
processing.  Those  pursuing  cloud  computing  should  focus  on  reaching  the  high-hanging 
fruits,  the  removal  of  unnecessary  constraints  surrounding  the  storage,  retrieval,  and 
manipulation  of  data-objects,  a  state  called  ‘cloud  Nirvana’.  In  cloud  Nirvana  any  object 
created  on  one  computing  device  (e.g.,  desktop,  smartphone,  tablet)  will  be  accessible 
from  any  another  computing  device  regardless  of  how  the  object  was  created  or  what 
application  is  opening  the  object  (Foster  et  ah,  2010a). 

Changing  to  a  data-centric  view  has  the  potential  to  fundamentally  change 
computing  just  as  other  past  innovations  have,  such  as:  modems,  laptops,  the  Internet, 
and  mobile  computing  changed  the  usage  of  the  technologies  of  their  respective  era.  For 
instance,  modems  by  allowing  multiple  users  to  utilize  a  mainframe  from  offsite,  laptops 
by  allowing  users  to  take  applications  and  data  with  them,  the  Internet  by  allowing  access 
to  email  and  other  corporate  software  from  anywhere,  and  mobile  computing  devices 
(e.g.,  smartphones,  tablets,  laptops)  by  allowing  users  to  access  repurposed  content  from 
internet  enabled  devices. 

While  documenting  a  roadmap  for  reaching  a  cloud  Nirvana  is  not  the  focus  of 
this  research,  it  is  important  to  introduce  the  initial  high-level  stages  (Migration, 
Integration,  and  Unification),  which  need  to  be  accomplished  before  Nirvana  can  be 
realized. 
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The  Migration  stage  involves  the  movement  of  existing  data  and 
applications  from  various  sources  to  the  cloud,  and  the  merging  into  an 
integrated  working  environment  (I WE). 

The  Integration  stage  aims  for  a  working  environment  that  provides  an 
integrated  tool  for  collaboration  and  communication  activities  and 
involves  merging,  or  purging,  of  unnecessary  duplication  of  both  data  and 
applications. 

In  the  Unification  stage  we  leverage  the  cloud  environment  to  the  ultimate 
level  of  collaboration  and  communication  -  the  unification  of  the 
workload.  The  final  boundaries  between  the  data  and  application  are 
removed  leading  to  an  information-centric  environment.  Rather  than 
having  data  and  applications  we  have  artifacts,  which  are  embodiments  of 
data  and  their  associated  manipulators,  mini  programs  that  allow  the  user 
to  process  (e.g.,  view,  edit,  and  print)  the  data  (Foster  et  al.,  2010a). 

For  additional  information  regarding  research  into  realizing  a  cloud  Nirvana, 
please  reference  the  article:  “Removing  the  Boundaries:  Steps  Toward  a  Cloud  Nirvana” 
in  the  August  2010  edition  of  IEEE  International  Conference  on  Granular  Computing 
(Foster  et  al.,  2010a). 

B.  USE  CASE  ANALYSIS  AND  METHODOLOGY 
1.  Definition  and  Overview 

Use  cases  are  useful  in  capturing  and  communicating  functional  requirements  by 
capturing  who  (actor)  does  what  (interaction)  with  the  system,  for  what  purpose  (goal), 
without  dealing  with  system  internals  (Malan  &  Bredemeyer,  2001).  A  complete  set  of 
use  cases  specifies  all  the  different  ways  to  use  the  system,  and  therefore  defines  all 
behavior  required  of  the  system,  bounding  the  scope  of  the  system. 

Use  cases  are  used  in  software  engineering  to  identify,  clarify,  and  organize 
system  functional  requirements  (intended  behavior  of  the  system)  in  an  easy  to 
understand  manner.  Generally,  use  case  scenarios  are  written  in  an  easy-to-understand 
structured  narrative  using  the  vocabulary  of  the  domain  (Malan  &  Bredemeyer,  2009). 
This  makes  it  easy  for  users  to  follow  and  validate  the  use  cases,  and  the  accessibility 
encourages  users  to  be  actively  involved  in  defining  system  requirements.  Use  cases  are 
initiated  by  a  user  with  a  particular  goal  in  mind  and  contain  all  system  activities  and 
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possible  scenarios  that  are  of  significance  to  the  users.  The  system  is  treated  as  a  “black 
box”  with  all  interactions,  including  responses,  being  perceived  as  coming  from  outside 
the  system. 

Use  cases  describe  the  sequence  of  interactions,  and  potential  variants  of  this 
sequence,  between  external  actors  and  the  system  under  consideration  necessary  to  satisfy 
the  goal.  Actors  are  parties,  outside  the  system,  that  interact  with  the  system  and  may  be  a 
class  of  users,  roles  users  can  play,  or  other  systems  (Cockbum,  2000). 

A  use  case,  or  set  of  use  cases,  has  the  characteristics  below  ("What  is  a  use 
case?,"  2008): 

•  Describes  one  main  flow  of  events,  and  possibly  variants  of  this  flow 

•  Is  multi-level,  so  that  one  use  case  can  use  the  functionality  of  another  one 

•  Models  the  goals  of  system/actor  (user)  interactions 

•  Organizes  functional  requirements 

•  Records  paths  (called  scenarios)  from  trigger  events  to  goals 
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III.  CURRENT  T&E  PROCESS 


A.  INTRODUCTION 

Tracking  and  maintaining  situational  awareness  of  documents,  presentations,  and 
other  types  of  business-driven  data  is  difficult  even  in  small  organizations  in  a  single 
location.  Naturally,  the  efficient  management  of  this  type  of  infonnation  becomes  more 
difficult  in  large  organizations  such  as  the  Anny  Test  and  Evaluation  Command  (ATEC), 
with  thousands  of  users  and  multiple  physical  locations.  From  a  program  management 
perspective,  inefficient  management  of  this  infonnation  leads  to  stale  data  upon  which  to 
base  decisions.  From  an  IT  management  perspective,  inefficient  management  of  data 
results  in  unnecessary  infrastructure  costs. 

This  chapter  provides  a  high-level  mission  scenario  of  the  current  process  for 
program  management,  test  reporting,  and  range-test  scheduling  within  ATEC.  We  first 
provide  an  overview  of  the  ATEC  domain,  followed  by  a  walkthrough  of  a  typical 
scenario  for  acquiring  and  conducting  live-fire  tests  of  a  system.  We  document  the 
scenario  by  using  use  cases,  activity,  and  collaboration  diagrams.  These  artifacts  are  used 
in  Chapter  IV  to  motivate  the  discussion  of  how  cloud  computing  in  its  current  form 
could  meet  the  needs  of  the  Anny  T&E  community.  The  selected  use  cases  were  chosen 
as  they  could  easily  be  extrapolated  to  other  domains,  as  every  DoD  domain  will  have 
program  management,  report  collaboration,  data  storage,  and  resource-scheduling 
requirements.  The  use  cases  in  Chapter  III  and  Chapter  IV  are  based  on  the  same  high- 
level  mission  scenario. 

B.  OVERVIEW  OF  THE  ARMY  T&E  COMMAND  DOMAIN 

1.  Background 

Title  10  of  the  U.S.  Code  requires  that  major  acquisition  programs  undergo 
independent  operational  test  and  evaluation  (OT&E)  in  order  to  proceed  beyond  low-rate 
initial  production  (LRIP)  ("DoD  Instruction  5000.02",  2010).  The  intent  of  the  law  is  to 
establish  a  system  of  checks  and  balances  that  will  ensure  that  soldiers  in  the  field  are 
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equipped  with  the  best  possible  equipment  (Johnson,  2003).  The  three  U.S.  military 
departments  have  all  undertaken  initiatives  to  link  the  tests  of  systems  under  development 
with  the  operational  tests  that  involve  test  exercises  in  the  field.  To  facilitate  this 
initiative,  the  Army  reorganized  its  T&E  program  in  October  1999  to  place 
developmental  and  operational  testing  under  one  command,  ATEC. 

ATEC  plans,  conducts,  and  integrates  developmental  testing,  and  independent 
operational  testing,  independent  evaluations,  assessments,  and  experiments  in  order  to 
provide  essential  information  to  acquisition  decision  makers.  ATEC  is  comprised  of  three 
subordinate  commands:  Developmental  Test  Command  (DTC),  Army  Evaluation  Center 
(AEC),  and  Operational  Test  Command  (OTC)  each  with  a  different  focus  (“U.S.  Army 
Test  and  Evaluation  Command,”  2010)  see  Figure  13. 


ARMY  TEST  AND  EVALUATION  COMMAND  IATEC) 


Integrated 

Operational 

Developmental 

Evaluations 

Testing 

Testing 

Figure  13.  Army  T&E  Domain  from  (“U.S.  Army  Test  and  Evaluation  Command,” 

2010) 
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a. 


Developmental  Test  Command  (DTC) 


DTC  is  the  technical  tester  for  the  ATEC.  DTC  tests  military  hardware 
and  software  of  every  description  under  precise  conditions  across  the  full  spectrum  of 
arctic,  tropical,  desert,  and  other  natural  or  controlled  environments.  DTC  provides  a  full 
range  of  test  services,  including  providing  unbiased  test  data  on  the  technical  feasibility 
of  early  concepts,  detennining  system  performance  and  safety,  assessing  technical  risks 
during  system  development,  confirming  designs  and  validating  manufacturers’  facilities 
and  processes  at  both  system  and  component  levels.  DTC  offers  its  testing  capabilities  to 
all  U.S.  military  services  and  DoD  organizations,  along  with  federal  agencies,  state  and 
local  governments,  foreign  and  allied  govermnents  and  private  industry  (“U.S.  Army 
Developmental  Test  Command,”  2010). 

b.  Operational  Test  Command  ( OTC ) 

Following  developmental  testing  and  issue  of  a  safety  release,  a  piece  of 
equipment  is  delivered  to  the  OTC  for  independent  OT&E.  OTC’s  mission  is  to  conduct 
realistic  operational  testing  with  representative  soldiers  and  units  in  the  areas  of 
equipment,  doctrine,  force  design,  and  training.  To  perform  this  mission  OTC  plans, 
conducts,  and  reports  operational  tests,  assessments,  and  experiments  in  order  to  provide 
essential  information  for  the  acquisition  and  fielding  of  warfighting  systems.  Ultimately 
OTC  is  responsible  for  ensuring  each  piece  of  equipment  is  operationally  tested  before 
being  placed  in  the  hands  of  warfighters  (“U.S.  Army  Operational  Test  Command,” 
2010). 


c.  Army  Evaluation  Center  (AEC) 

AEC  provides  the  evaluation  portion  of  ATEC’s  T&E  mission.  AEC  plans 
and  conducts  independent  evaluations  and  assessments  of  acquisition  programs  providing 
the  results  to  DoD  decision  makers  and  soldiers.  AEC  evaluates  the  data  obtained  from 
DTC,  OTC,  contractor  testing,  and  modeling  and  simulation  (M&S)  events  to  determine 
a  system’s  operational  effectiveness,  suitability,  and  survivability.  AEC  is  the 
organization  that  writes  the  final  report,  System  Evaluation  Report  (SER),  used  by  the 
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decision  makers  to  determine  whether  a  new  or  enhanced  system  will  be  fielded  and 
become  part  of  the  Army’s  arsenal  (“U.S.  Army  Evaluation  Center,”  2010). 

2.  ATEC  Enterprise 


•Request  for  Test  Services 

•  Models  &  Simulation s  (pretest  plan n i n g.  syn  th etic  env) 

* 

t 

* 

t 

Financial 

Test  Technology 

T&E  Mgmt 

Document  & 

Management 

Planning 

EVM 

Test  Data  Library 

Figure  14.  ATEC  Enterprise  Notional  High-Level  System  View  (circa  2006) 


Within  the  ATEC  Enterprise  (Figure  14),  there  is  a  need  for  a  consolidated 
program  management  and  test  reporting  system.  Currently,  ATEC  and  its  subordinate 
commands  have  numerous  stovepipe  systems  and  workflow  processes,  which  at  best, 
awkwardly  communicate  program  status  to  customers  and  ATEC  leadership.  Through  the 
use  of  a  private  cloud,  the  current  process  could  be  replaced  with  a  more  automated 
process  allowing  customers,  leadership,  and  other  stakeholders  to  pull  information  as 
needed  in  an  automated  fashion  rather  than  manually  contacting  the  local  test  center  point 
of  contact  (POC)  to  obtain  that  infonnation.  Systems,  such  as  the  Standard  Operating  and 
Maintenance  Army  Research  and  Development  System  (SOMARDS),  SOMARDS 
Financial  Information  Management  System  (SOFIMS),  ATEC  Decision  Support  System 

(ADSS),  Versatile  Information  Systems  Integrated  On-Line  (VISION)  Digital  Library 
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System  (VDLS),  and  Performance  Measurement  Enterprise  System  (PMES)  are  all 
involved  in  the  current  program  management  and  test  report  generation/delivery  process, 
as  depicted  in  Figure  15.  A  description  of  each  of  these  systems  follows. 

a.  SOMARDS 

SOMARDS  is  the  Army’s  authoritative  financial  management  and 
reporting  system  that  maintains  all  actual  financials  and  is  used  by  the  Army  to  collect, 
manage,  and  distribute  funds.  It  provides  for  reimbursable  customer  and  direct  mission 
funds  control,  reporting  for  labor,  reimbursable  billings,  advances,  and  general  operating 
expenses.  Particular  features  include:  on-line  and  batch  processing;  general  ledger 
reporting;  production  of  daily,  regulatory,  and  monthly  reports;  file  inquiry/maintenance 
capability;  and  month-end/year-end  and  purge  processes. 

b.  SOFIMS 

SOFIMS  is  an  internal  ATEC  system  that  performs  daily  imports  from 
SOMARDS.  SOFIMS  restructures  SOMARDS  information  from  each  update  in  order  to 
present  infonnation  in  a  format  that  is  easier  to  use  by  ATEC  customers  than 
SOMARDS.  It  provides  financial  execution  information,  commitments,  obligations, 
costs,  and  disbursements  relative  to  both  direct  and  reimbursable  dollars  received  by  the 
command.  It  allows  users  to  store,  sort,  search,  and  present  command- wide  financial 
information  in  a  user-customizable  manner. 

c.  ADSS 

ADSS  is  ATEC’s  Web  based  T&E  planning  tool  that  tracks  the  resources 
(e.g.,  equipment,  labor,  dollars)  that  are  required  to  conduct  a  test  or  evaluation.  It  is  used 
as  a  comprehensive  management  system  to  track,  record,  and  monitor  T&E  planning, 
execution,  and  reporting  for  all  ATEC  T&E  activities.  It  contains  all  system  T&E 
information  including  ATEC  System  Team  (AST)  membership,  program  schedules,  T&E 
schedules,  cost  estimates,  funding  requirements,  and  document  status  and  dates  for  all  of 
ATEC’s  T&E  projects.  ADSS  is  the  official  repository  for  tracking  all  Developmental 
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Test  (DT),  and  Operational  Test  (OT)  projects  that  identify  resources  (e.g.,  units, 
personnel,  ranges,  equipment)  for  tests  requiring  soldier  support.  ADSS  also  provides  a 
single  source  for  requesting  DTC  test  services  and  provides  its  own  estimated  financial 
data  by  pulling  in  actual  financials  from  SOFIMS  on  a  daily  basis. 

d.  VDLS 

VDLS  is  a  Web-based  knowledge  management  system  that  provides 
distributed  information  management  capability  supporting  information  fusion.  VDLS 
provides  access  control  so  that  information  is  protected  and  provided  to  only  those  users 
with  the  required  access  permissions.  VDLS  provides  users  a  place  to  store  and  access 
information  from  any  place  that  has  an  Internet  connection.  VDLS  is  ATEC’s  primary 
business  tool  for  the  collection,  management,  and  timely  dissemination  of  data  and 
information  for  decision  making.  When  an  ADSS  test  project  becomes  active  a  test 
project  shell  is  created  within  VDLS.  The  project  shell  is  the  mandated  repository  for  the 
interim  data  report,  final  test  report,  test  record,  and  all  level-three  and  above  test  data.  A 
repository  for  level-one  and  level-two  data  does  not  currently  exist;  for  a  description  of 
the  seven  different  levels  of  T&E  data  see  Table  10  in  Chapter  IV.B. 

<?.  PMES 

PMES  provides  the  ability  to  retrieve  and  display  accrual  data  directly  in 
Microsoft  Project  for  analysis  and  reporting  purposes.  If  this  infonnation  is  updated 
regularly,  users  should  be  able  to  analyze  spending  patterns  over  time  in  the  context  of  a 
project’s  estimated  spending  plan.  ADSS  provides  the  central  repository  and  other 
functionality,  which  allows  PMES  to  serve  as  a  single  T&E  program  management 
system.  Currently,  PMES  is  only  deployed  to  a  few  ATEC  pilot  locations  so  its  usage  is 
not  widespread. 

C.  TYPICAL  T&E  MISSION  THREAD 

A  program  is  a  set  of  projects  that  are  managed  together  to  achieve  a  common 
goal.  Programs  are  typically  large  undertakings  that  require  integrated  management  of  the 
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individual  projects  that  make  up  the  program.  A  project  is  a  temporary  endeavor,  having 
a  defined  beginning  and  end.  It  is  undertaken  to  meet  unique  goals  and  objectives  in  order 
to  deliver  specific  outputs  usually  bringing  about  beneficial  change  or  added  value 
(Nokes,  2008).  An  individual  DTC,  AEC,  or  OTC  weapon  test  could  be  considered  to  be 
a  project  with  the  delivered  product  being  the  respective  final  test  report.  The  entire 
coordinated  set  of  DTC,  AEC,  and  OTC  projects  for  a  specific  weapons  platform 
comprise  a  program.  At  ATEC,  the  typical  system  evaluation  is  structured  as  a  program 
with  AEC,  DTC,  and  OTC  each  managing  their  respective  pieces  but  in  coordination 
with  each  other. 

Program  management  is  the  discipline  of  planning,  organizing,  and  managing 
resources  to  bring  about  the  successful  completion  of  specific  project  goals  and 
objectives,  as  shown  in  Figure  15.  Program  management  emphasizes  the  coordination 
and  prioritization  of  resources  across  projects,  managing  links  between  the  projects,  and 
managing  the  costs  and  risks  of  the  program. 


Initiation 


Planning  and 
Design 


?  % 


Executing 


k-> 


Monitoring 
and  Controlling 


Closing 


Figure  15.  Program  Management  Steps  from  (“Management  Steps  Part  3,”  2009) 


ATEC  does  not  currently  have  a  comprehensive  standard  way  of  performing 
program  management.  Program  managers  (PM)  are  not  able  to  properly  monitor  the 
current  status  of  projects  within  their  program.  As  a  result  the  program  manager,  and 
ultimately  ATEC  senior  leadership,  may  be  making  decisions  based  on  incomplete  or 
outdated  information. 
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The  current  system  provides  monitoring  at  the  milestone  level,  which  only  reflects 
a  snapshot  in  time  of  the  project  status.  Utilizing  external  task  relationships  and 
consolidated  projects  with  Microsoft  Project  enables  the  program  manager  or  AST  Chair 
to  coordinate  and  view  a  program  schedule.  However,  the  current  implementation  does 
not  allow  the  PM  to  drill  down  to  the  actual  range  schedule,  as  only  the  milestones  are 
currently  available.  This  is  due  to  the  multiple  semi-stovepipe  systems  that  comprise  the 
ATEC  Enterprise,  as  shown  in  Figure  14. 
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Figure  16.  ADSS  SOFIMS  SOMARDS  Integration 


The  ATEC-specific  SOFIMS  pulls  financial  information  from  SOMARDS 
(Figure  16).  Test  milestone  and  rudimentary  schedule  information  is  stored  in  ADSS  with 
milestones  being  pushed  into  SOFIMS.  Level  3-plus  data — test  reports  of  record — are 
stored  within  VDLS  in  a  folder  structure  based  on  the  unique  ADSS  number  and 
milestones  within  the  ADSS  schedule.  All  of  the  previously  mentioned  tools  are 
primarily  used  at  the  ATEC  HQ  level  for  tracking  expenditures  and  the  health  of  a 
program  based  on  high-level  goals.  However,  at  the  test  centers  (TC)  where  the  testing 
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actually  occurs  there  are  more  specific  tools,  such  as:  multiple  unique  range  scheduling 
systems  that  are  used  by  range  personnel  to  schedule  tests,  deconflict  safety  fans, 
deconflict  air  or  ground  space,  and  track  meta-data  about  the  test,  such  as  serial  numbers 
and  test  setup  information.  Also,  at  the  local  test  centers  there  are  data  storage  systems 
that  store  everything  from  the  raw  collected  test  data  to  the  final  level-three  test  report. 

A  safety  fan  refers  to  the  surface  danger  zone  (SDZ)  for  the  SUT.  SDZs  represent 
the  minimum  safety  boundary  surrounding  a  live  fire  test  event,  and  are  unique  for  each 
range  and  SUT.  “The  objective  of  a  SDZ  is  to  represent  the  residual  risks  of  fragment 
escapes  or  other  danger  to  the  public  at  no  greater  than  10'6  (one  in  a  million)”  (Anny, 
2003). 


1.  Program  Management 

After  completing  a  test  for  an  evaluated  system  at  any  of  the  ATEC  TCs,  a 
process  similar  to  the  one  shown  in  Figure  17  and  the  corresponding  collaborations 
shown  in  Figure  18  occur.  For  non-evaluated  systems  the  process  is  a  subset  of  the 
evaluated  system. 
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Figure  17.  Program  Management  Process 
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Figure  18.  Program  Management  Collaborations 


The  primary  scenario  used  throughout  the  remainder  of  this  research  deals  with  an 
evaluated  system,  although  a  non-evaluated  system  would  utilize  a  subset  of  the  same 
processes.  Prior  to  full  production  of  a  new  weapons  platform  a  PM  decides  to  split  LRIP 
between  multiple  contractors  for  the  mandated  Live  Fire  Test  and  Evaluation  (LFT&E) 
portion  of  the  test  program.  This  is  also  known  as  a  contractor  fly-off  and  is  done  in  an 
effort  to  obtain  the  best  value  choice  for  the  government.  A  fly-off  allows  the  government 
to  keep  competitive  forces  working  for  a  longer  period  of  time  and  make  final  decisions 
based  on  information  that  is  as  realistic  as  possible.  It  provides  the  government  an 
opportunity  to  reduce  the  risk  that  a  single  source  contractor  will  not  deliver  a  product 
that  meets  the  requirements  as  the  government  can  pick  the  best  delivered  LRIP  product 
based  on  hard  requirements  and  evaluation  criteria. 

When  an  external  customer  or  ATEC  organization  requires  ATEC  services,  the 
responsible  party  submits  a  Request  for  Test  Services  (RFTS)  to  formally  initiate  the 
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request.  The  PM  submits  a  Request  for  Test  Services  (RFTS)  to  ATEC  to  establish  a  test 
program.  After  the  RFTS  has  been  submitted,  ATEC  will  Initiate  the  request,  creating  an 
ADSS  Effort  Shell  (Effort),  and  then  forward  it  to  the  appropriate  DTC  Test  Manager 
(TM).  The  final  step  in  the  ATEC  project  initiation  process  is  to  create  a  plan  in 
Microsoft  Project  to  support  detailed  planning  and  estimation.  This  project  plan  will  be 
created  using  PMES  and  the  initial  project  plan  will  contain  all  of  the  milestones  and 
other  infonnation  that  was  accumulated  during  the  previous  steps. 

The  DTC  TM  then  detennines  which  TC  is  best  suited  to  accomplish  the  testing 
requirements,  activates  the  effort,  and  forwards  the  request  to  the  selected  TC. 
Throughout  the  request  distribution  process,  DTC  HQ,  DTC  TM,  and  the  TC  may  add 
and  edit  milestone  infonnation  contained  within  the  effort.  Upon  receipt  of  the  effort,  the 
TC  assigns  a  Test  Director  (TD)  to  coordinate  all  local  tests,  generate  a  test  plan,  and 
manage  the  effort.  This  assignment  is  officially  recorded  within  ADSS  in  the  TC  POC 
field. 

2.  Conduct  Test  Process 

The  TD  then,  based  on  the  initial  test  plan  requirements,  distributes  and 
coordinates  the  various  stages  of  testing  to  the  TC  functional  directorates  (e.g..  Missile 
Perfonnance  and  Sensors,  Aviation,  Environmental  and  Component)  where  a  Test 
Engineer  (TE)  is  selected  to  actually  conduct  and  manage  the  day-to-day  status  of  testing. 
Each  selected  TE  is  responsible  for  either  performing  or  coordinating  all  aspects  of  their 
portion  of  the  test  plan,  covering  everything  from  pretest  test  planning,  test  execution, 
post  test  analysis  and  data  verification,  to  collaboratively  authoring  the  final  test  report 
(TR). 

a.  Pretest  Setup  Process 

Pretest  coordination  is  necessary  because  conducting  a  test  is  potentially  a 
destructive  process  and  costly  process.  The  coordination  is  done  for  safety  and  to  ensure 
that  all  test  data  is  captured  the  first  time,  every  time.  That  way  the  test  does  not  have  to 
be  repeated  at  a  later  date  due  to  poor  planning.  Figure  19  shows  the  pretest  planning  and 
setup  process. 
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Figure  19.  Pretest  Setup  Process 


During  pretest  planning,  the  TE  verbally  socializes  the  system  under  test 
(SUT)  anticipated  test  dates  with  other  co-located  TEs  and  range  personnel  to  determine 
if  the  dates  will  cause  any  unavoidable  conflicts.  The  TE  also  works  with  Instrumentation 
Support  Personnel  (ISP)  to  determine  if  resources  (e.g.,  hardware,  instrumentation, 
people)  will  be  available  during  the  anticipated  dates.  Once  the  TE  is  satisfied  there  are 
no  obvious  roadblocks  then  the  TE  will  go  through  the  process  of  creating  a  schedule 
request. 

The  TE  then  begins  working  with  the  ISP  to  identify  which  ground  truth 
data  elements  the  TE  will  need  to  evaluate  the  SUT.  Ground  truth  data  refers  to 
measurements  collected  from  properly  calibrated  instrumentation  used  as  a  means  of 
providing  a  baseline  “reality”  vs.  what  the  SUT  “perceives”  reality  to  be.  The  ISP  will 
use  that  information  to  determine  what  instrumentation  is  required  for  the  test,  and  where 
it  should  be  placed  to  capture  the  required  ground  truth  data  (Figure  20).  The  TE 
documents,  the  instrumentation  setup  information,  what  we  call  in  this  thesis,  the 
Instrumentation  Configuration  Plan. 
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Figure  20.  Pretest  Setup  Collaborations 


b.  Schedule  Test  Process 

Currently,  a  standard  tool  does  not  exist  within  ATEC  for  scheduling 
range  time  or  for  identifying  and  resolving  resource  conflicts.  Most  test  ranges  have  some 
type  of  home-grown  application  that  performs  parts  of  the  scheduling  deconfliction 
function.  These  tools  merely  enable  more  efficient  communications  between  the  parties 
that  share  resources  and  in  most  cases  these  collaborations  will  occur  weeks  prior  to  the 
SUT  test  date.  If  ranges  do  not  have  a  tool  to  assist  in  their  long-range  and  short-term 
planning  and  conflict  resolution,  range  personnel  utilize  a  manual  process  similar  to  the 
one  depicted  in  Figure  2 1 . 
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Figure  2 1 .  Schedule  Test  Process 


At  a  range  when  a  TE  desires  to  put  a  test  on  the  official  authoritative 
schedule,  several  things  must  first  occur.  For  example,  a  test  location  must  be  selected, 
safety  fans  must  be  established  and  evaluated,  coordination  must  occur  between  this  test 
and  any  conflicting  tests,  frequency  authorizations  (if  needed)  must  be  obtained,  and 
spatial  areas  must  be  coordinated  with  any  other  scheduled  tests,  ranges,  and  potentially 
the  Federal  Aviation  Administration  (FAA)  (via  a  Notice  to  Ainnen  (NOTAM)  post.) 
This  information  is  used  by  the  home-grown  scheduling  tools  or  by  individuals  with 
grease  or  white  boards  to  identity  potential  conflicts. 

c.  Event  Deconfliction  Process 

LFT&E  activities  cover  a  wide  range  of  operations,  such  as  manned  flight, 
unmanned  flight,  manned  ground  vehicles,  unmanned  ground  vehicles,  static  ordnance 
tests,  ordnance  flight  operations,  climatic  environmental  tests,  electromagnetic 
environment  tests,  road  tests  and  operator  training.  Within  a  single  TC,  two  or  more  of 
these  activities  regularly  occur  simultaneously  and  require  coordination  of  shared 

resources  such  as  radio  frequencies,  and  spatial  area  (i.e.,  air,  water,  and  ground  space). 
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The  deconfliction  process  typically  takes  place  weeks  prior  to  the  requested  test  dates 
because  the  longer  a  TE  waits  to  request  time  on  the  authoritative  schedule  the  more 
likely  the  TE  will  have  conflicts. 


This  scheduling  event  deconfliction  process  (Figure  22  and  Figure  23)  is 
an  iterative  process  that  repeats  until  all  conflicts  are  either  resolved  or  the  schedule 
request  is  modified.  Resolved  means  that  either  the  conflicting  SUT  test  date  and  time 
has  been  modified  so  that  the  conflicting  test  events  can  both  occur,  or  that  manual 
resolutions  will  need  to  occur  prior  to  the  SUT  test  occurring.  Modifying  the  requested 
date  and  time  is  as  simple  as  it  sounds  and  will  result  in  the  TE  going  through  the 
schedule  test  process  and  starting  the  event  deconfliction  process  all  over  again  because 
the  change  could  result  in  new  conflicts.  Manually  resolving  resource  conflicts  can  be 
time-consuming  and  error  prone. 

SUT  test  Alpha  is  scheduled  for  the  same  date  and  time  as  SUT  test  Beta 
and  Alpha’s  safety  fan  barely  overlaps  with  the  safety  fan  for  Beta  causing  a  conflict.  The 
responsible  TEs  could  determine  after  consulting  with  the  range  safety  officer  and  the 
standard  operating  procedures  for  both  SUTs  that  both  tests  can  stay  on  the  schedule. 
However,  a  manual  resolution  is  notated  that  Alpha  must  get  positive  confirmation  via 
radio  from  the  lead  Beta  POC  that  all  Beta  test  personnel  have  exited  the  safety  fan  area 
prior  to  Alpha  conducting  its  test. 
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Figure  22.  Event  Deconfliction  Process 
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Figure  23.  Event  Deconfliction  Collaborations 


3.  Test  Execution  Process 

After  the  TE  has  successfully  planned  for  and  scheduled  the  test  event,  the  TC 
range  hosts  each  of  the  competing  contractors  and  conducts  testing  on  each  SUT 
individually.  Everything  up  until  this  point  has  been  equivalent  to  a  dress  rehearsal  for  a 
play  with  each  actor  rehearsing  his  or  her  lines  and  movements.  During  the  test  execution 
(Figure  24),  the  Instrumentation  Configuration  Plan  is  executed,  all  test  setup  information 
is  documented  for  archival  purposes  (in  case  the  test  needs  to  be  performed  again  in  the 
future),  the  test  plan  is  executed,  the  test  occurs  (e.g.,  missile  is  fired,  vehicle  is  shaken, 
aircraft  is  flown,  explosive  is  detonated),  and  ground  truth  data  is  collected  (Figure  25). 
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Figure  24.  Test  Execution  Process 
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Figure  25. 


Test  Execution  Collaborations 


4.  Post-test  Data  Analysis  Process 

During  testing,  a  large  amount  of  raw  data  and  ground  truth  data  is  generated. 
This  data  must  be  collected,  stored,  and  secured  for  analysis  and  reduction.  At  the 
conclusion  of  the  test,  the  TE  is  responsible  for  coordinating  with  all  involved 
Instrumentation  Support  Personnel  to  gather  collected  raw  test  data  and  sensor  ground 
truth  data  (Figure  26).  The  ISP  will  collect  the  data  from  the  various  instruments,  sensors, 
and  perhaps  even  the  SUT  itself.  Typically,  this  raw  data  is  stored  on  a  local  server  and 
can  consist  of  everything  from  small  binary  files,  high-resolution  photo  and  high-speed 
video  files,  to  high-fidelity  Time  Space  and  Position  Information  (TSPI)  data.  The  TE 
will  then  be  notified  by  some  mechanism,  typically  verbal  or  e-mail,  that  all  information 
has  been  collected  and  stored  at  the  central  range  location. 
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Figure  26.  Post-test  Data  Analysis  Process 


The  TE  is  responsible  for  setting  permissions  on  these  files,  which  is  paramount 
as  at  all  times  the  competing  contractors  should  not  be  allowed  to  see  any  information 
regarding  their  competition  or  even  know  that  a  test  has  been  conducted  on  their  product. 
For  example,  if  contractor  “Alpha”  were  to  inadvertently  gain  access  to  information 
regarding  competing  contractor  “Beta”  then  this  could  give  “Alpha”  an  unfair  advantage. 
Even  knowing  the  number  of  times  “Beta”  has  been  at  a  test  range,  the  number  of  flights 
performed  by  “Beta,”  or  even  the  fact  that  “Beta”  has  flights  on  a  test  range’s  upcoming 
schedule  could  potentially  be  used  by  “Alpha”  to  determine  if  “Beta”  is  having  problems 
in  their  development.  If  “Beta”  ever  discovered  that  “Alpha”  had  access  to  such 
information,  and  “Alpha”  won  the  eventual  contract  award,  it  could  lead  to  a  costly 
contract  protest. 

After  the  data  has  been  properly  secured  on  the  server,  the  TE  then  begins  the 
process  of  verifying  that  there  were  no  anomalies  in  the  collected  ground  truth  data  and 
begins  reducing  the  collected  test  dataset  to  contain  only  the  information  pertinent  to  the 
test  objectives  (Figure  27).  This  reduced  dataset  is  typically  stored  both  on  the  server  and 
also  temporarily  stored  on  the  TE’s  local  machine.  Ultimately,  all  of  the  reduced  data  is 
used  by  the  TE  to  generate  the  final  TR. 
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Figure  27.  Post-test  Data  Analysis  Collaborations 


a.  Test  Report  Generation  Process 

Everything  described  within  Chapter  III.C,  from  the  initial  Program 
Management  through  the  various  steps  of  Pretest  Planning,  Scheduling,  and 
Deconfliction  to  Conducting  the  Test,  has  been  leading  up  to  the  generation  of  the  final 
TR.  The  process  that  started  in  Chapter  III.C.  1  culminates  with  the  delivery  of  the  final 
TR  to  the  PM  (Figure  28). 
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Figure  28.  Test  Report  Generation  Process 


Test  Reports  (TR)  document  the  results  of  a  test  and  usually  recommend  a 
course  of  action  based  on  those  results.  The  TR  addresses  test  results,  including  test 
conducted,  data  collected,  the  data  reduction  and  analysis  process,  and  conclusions  to  be 
drawn  from  the  test  data.  Overall,  it  provides  data  on  the  T&E  activity  completed  during 
the  engineering  and  development  phase  of  a  program  to  verify  that  the  hardware  and 
software  design  meets  the  specified  requirements. 

DoD  Instruction  5000.02  requires  the  PM  of  a  program  designated  for  the 
Office  of  the  Secretary  of  Defense  (OSD)  T&E  oversight  to  provide  reports  of  results, 
conclusions,  and  recommendations  from  Developmental  Test  and  Evaluation  (DT&E), 
OT&E,  and  LFT&E  to  the  Director,  Operational  Test  and  Evaluation  (DOT&E)  and 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (USD(AT&L)) 
(“DoD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System,”  2010).  For 
those  reports  supporting  a  decision  point,  the  report  will  identify  the  strengths  and 
weaknesses  in  meeting  the  warfighters,  documented  requirements  based  on 
developmental  evaluations.  The  content  of  the  report  is  the  impartial  evaluation  from  the 
DT&E  Responsible  Test  Organization  (RTO)  of  a  system’s  military  utility  and 
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capabilities  against  warfighter  requirements.  The  DT&E  report  will  provide  a  historical 
record  of  the  final  T&E  results  for  the  system.  In  addition,  the  TR  should  include  the 
RTO’s  assessment  of  a  system’s  military  utility,  capabilities  and  limitations;  document 
the  test  techniques,  procedures,  and  data  analysis;  and  provide  data  for  operating  and 
employment  and  maintenance  manuals  for  the  system.  At  the  conclusion  of  LFT&E,  the 
DOT&E  prepares  an  independent  assessment  report  that  describes  the  results  of  the 
survivability  or  lethality  LFT&E,  and  assesses  whether  the  LFT&E  was  adequate  to 
provide  information  to  decison  makers  on  potential  user  casualties  and  system 
vulnerabilities  or  lethality  is  based  on  testing  under  realistic  conditions,  consideration  of 
the  validated  statement  of  desired  operational  capabilities,  the  expected  threat,  and 
susceptibility  to  attack  (“Defense  Acquisition  Guidebook  Ch.  9.7.  Test  and  Evaluation 
Reporting  of  Results,”  2010). 

Figure  29  shows  the  typical  collaborations  between  the  TE  and  the  TM 
prior  to  a  final  TR  being  delivered  to  a  PM.  After  the  TE  has  reviewed  and  reduced  all 
data  collected  during  the  test,  then  the  TE  prepares  a  draft  TE  TR.  Prior  to  the  draft  TE 
TR  being  finalized,  it  must  go  through  an  iterative  document  collaboration  and  review 
processes  with  the  TE  Supervisor  after  which  it  is  delivered  to  the  TD  where  it  is 
combined  with  any  other  TE  TRs  that  were  part  of  the  same  main  test  for  the  TC.  This 
combined  report,  after  going  through  an  iterative  document  collaboration  and  review 
process  discussed  in  the  next  section,  will  become  the  final  TC  TR  and  will  be  uploaded 
to  the  VDLS  location  that  was  previously  created. 

The  internal  TC  portion  of  the  process  described  above  can  take  ninety 
plus  days  to  complete  and  during  this  time  it  is  highly  likely  that  while  collaborating  on 
the  final  TR  the  TE,  supervisor,  or  TD  will  be  sent  on  a  temporary  duty  assignment 
(TDY).  This  leads  to  configuration  control  issues  as  the  person  going  TDY  must 
download  a  copy  of  the  current  version  of  the  document  from  the  server,  make  changes 
while  TDY,  and  upon  returning  must  manually  deconflict  any  changes  that  were  made  to 
the  document  by  the  other  reviewers. 
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After  the  final  TC  TR  has  been  uploaded  to  VDLS,  the  TM  will  then  be 
notified  that  the  final  TC  TR  has  been  uploaded  to  VDLS.  The  TM  will  then  collect  the 
final  TC  TRs  from  all  TCs  that  were  involved  in  the  overall  test.  The  TM  will  then  go 
through  an  iterative  document  collaboration  and  review  process  to  create  the  final  TR. 
This  final  TR  is  the  report  described  in  DoD  instruction  5000.02  and  will  be  sent  to  the 
PM. 
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Figure  29.  Test  Report  Generation  Collaborations 

5.  Document  Collaboration  Process 

When  a  document  enters  the  document  collaboration  process,  as  illustrated  in 
Figure  30,  it  goes  through  the  same  abstract  process  regardless  of  where  the  document 
originates  from  or  who  it  will  ultimately  be  delivered  too  and  involves  contributors  and 
reviews.  The  contributors  send  the  draft  document  to  the  reviewers  who  review  the 
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document  and  either  approve  it  or  don’t  approve  it.  If  the  draft  is  approved,  then  the 
draft’s  nomenclature  is  changed  from  draft  to  final.  If  the  draft  is  not  approved,  then  the 
reviewers  will  send  comments  back  to  the  contributors.  The  contributors  will  then  make 
the  requested  changes  and  resubmit  a  new  draft  starting  the  entire  process  over  again. 
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Figure  30.  Document  Collaboration  Process 


D.  USE  CASE  ANALYSIS  SUMMARY 

Figure  31  presents  a  Use  Case  model  for  the  typical  T&E  mission  thread 
described  in  Chapter  III.C,  which  is  made  up  of  nine  use  cases  described  in  Tables  1-9  in 
this  section. 
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Figure  3 1 .  Mission  Thread  Scenario 
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Program  Management 

Description 

A  Program  Manager  (PM)  requests  a  new  LFT&E  test. 

Desired 

outcome 

A  test  program  is  initiated  with  the  test  requirements  flowing  from  the 

PM  to  the  appropriate  TC  and  eventually  to  the  appropriate  TE  for  the 
test. 

Assumptions 

None 

Actors 

•  Program  Manager  (PM) 

•  Supervisor 

•  System  Under  Test  (SUT) 

•  Test  Center  (TC) 

•  Test  Director  (TD) 

•  Test  Engineer  (TE) 
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•  Test  Manager  (TM) 

Dependencies 

None 

Process  flow 

Step  description 

Artifact 
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8.  TD  gathers  requirements 

9.  TD  generates  Test  Plan 

Initial  Test  Plan  created 

ADSS  Milestones  updated 

VDLS  Folder  Structure  created 

10.  TD  selects  Division 

1 1 .  Division  Supervisor  selects  TE 

Deliverables 

•  ADSS  Effort 

•  Initial  Test  Plan 

•  Project  Plan 

•  Test  Requirements 

Additional 

infonnation 

None 

Table  1.  Use  Case  1:  Program  Management 
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UC2 

Conduct  Test 

Description 

Using  the  requirements  and  initial  test  plan  a  test  will  be  conducted.  This 

will  require  pretest  planning,  scheduling  coordination,  event 

deconfliction,  test  execution,  and  post  test  data  analysis. 

Desired 

To  successfully  conduct  the  test,  obtain  the  data  elements  that  will  allow 

outcome 

the  TE  to  properly  evaluate  the  SUT,  and  reduce  the  data  in  preparation 

for  the  generation  of  a  fonnal  Test  Report. 

Assumptions 

•  Funding  has  arrived,  safety  fans  have  been  created,  and 

environmental  impact  concerns  have  been  addressed. 

•  No  external  issues  exist  to  prevent  test  from  occurring. 

•  System  Under  Test  (SUT)  is 

onsite  at  the  Test  Center  (TC) 

Actors 

•  Instrumentation  Support  Personnel  (ISP) 

•  Test  Engineer  (TE) 

•  System  Under  Test  (SUT) 

Use  cases 
involved 

•  Post-test  Data  Analysis  UC7 

•  Pretest  Setup  UC3 

•  Test  Execution  UC6 

Dependencies 

•  Initial  Test  Plan 

•  Test  Requirements 

Process  flow 

Step  description 

Artifact 

1 .  Pretest  setup 

Finalized  test  plan 

2.  Schedule  test 

Test  on  authoritative  schedule 

3.  Test  execution 

Raw  test  data  collected 

4.  Post  test  data  analysis 

Test  data  reduced 

Deliverables 

•  Finalized  Test  Plan 

•  Scheduled  Test 

•  Raw  Test  Data 

•  Reduced  Test  Data 

Additional 

infonnation 

None 

Table  2.  Use  Case  2:  Conduct  Test 


66 


UC3 

Pretest  Setup 

Description 

Prior  to  test  execution  a  great  deal  of  upfront  planning  must  occur.  The 
test  plan  must  be  finalized,  instrumentation  must  be  configured, 
schedules  must  be  coordinated,  the  test  must  be  added  to  the  authoritative 
schedule,  and  test  setup  information  must  be  fully  documented. 

Desired 

outcome 

To  perform  the  proper  upfront  planning  and  coordination  for  the  test 
execution  to  be  repeatable  and  be  executed  with  as  few  oversights  as 
possible. 

Assumptions 

•  Test  Plan  has  been  finalized. 

Actors 

•  Instrumentation  Support  Personnel  (ISP) 

•  Schedule  Test  Process 

•  Scheduler 

•  Test  Engineer  (TE) 

Use  cases 
involved 

•  Schedule  Test  Process  UC4 

Dependencies 

•  Initial  Test  Plan 

•  Test  Requirements 

Process  flow 

Step  description 

Artifact 

1 .  Test  Engineer  requests 
range  time* 

Schedule  Request 

2.  Test  scheduled  by  Schedule 
Test  Process*  (UC4) 

Authoritative  Schedule  for  SUT 

3.  Notify  ISP  and  TE 

4.  ISP  configures 
instrumentation 

Instrumentation  Configuration  Plan 

5.  Store  Configuration  Plan 

Deliverables 

•  Schedule  Request 

•  Instrumentation  Configuration  Plan 

•  Authoritative  Schedule  for  SUT 

Additional 

infonnation 

*  Step  1  and  2  will  continue  will  continue  until  the  requested  schedule 
successfully  goes  through  the  Schedule  Test  Process  (UC4). 

Table  3.  Use  Case  3:  Pretest  Setup 


67 


UC4 

Schedule  Test 

Description 

Put  a  test  on  the  authoritative  range  schedule. 

Desired 

outcome 

Successfully  schedule  range  time  and  all  required  resources  needed  to 
support  a  test. 

Assumptions 

•  All  required  resources  are  known  and  identified  prior  to 
scheduling  test 

Actors 

•  Event  De-confliction  Process 

•  Scheduler 

•  Test  Engineer 

Use  cases 
involved 

•  Event  De-confliction  (UC5) 

Dependencies 

•  Final  Test  Plan 

Process  flow 

Step  description 

Artifact 

1 .  *  Schedule  Request 
received  from  Test 

Engineer 

Proposed  Request 

2.  *  Conflicts  identified  by 
Event  De-confliction 
process  (UC5) 

3.  Conflicts  resolved 

Test  Engineer  notified 

Test  added  to  Authoritative 

Schedule 

Deliverables 

•  Authoritative  Schedule  for  SUT 

Additional 

infonnation 

*  Step  1  and  2  will  continue  will  continue  until  all  conflicts  are 
addressed. 

Table  4.  Use  Case  4:  Schedule  Test 
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UC5 

Event  De-confliction 

Description 

Identify  and  resolve  potential  scheduling  conflicts  prior  to  test  events 
being  added  to  the  authoritative  schedule. 

Desired 

outcome 

To  have  a  fully  deconflicted  schedule,  so  there  is  not  a  schedule  delay 
due  to  lack  of  prior  coordination  between  test  programs. 

Assumptions 

None 

Actors 

•  Authoritative  Schedule  Information 

•  Conflictor 

•  Conflictor  Data 

•  Scheduler 

•  Test  Engineer 

Use  cases 
involved 

•  Schedule  Test  (UC4) 

Dependencies 

•  Proposed  schedule 

Process  flow 

Step  description 

Artifact 

1 .  Receive  proposed  schedule 

2.  Request  scheduled  events 

3.  Identify  conflicts 

List  of  conflicts 

4.  Send  conflict  list  to  user 

User  Notified 

Deliverables 

•  List  of  conflicts 

Additional 

infonnation 

If  no  conflicts  are  present  then  the  user  is  notified  that  no  conflicts  were 
found.  This  process  will  repeat  with  UC4  until  all  conflicts  are  addressed. 

Table  5.  Use  Case  5:  Event  De-confliction 
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UC6 

Test  Execution 

Description 

LFT&E  test  occurs  and  data  is  collected. 

Desired 

outcome 

The  test  plan  is  executed  with  no  problems  with  data  being  successfully 
collected,  stored,  and  secured  for  later  reduction  by  the  Test  Engineer 
(TE). 

Assumptions 

•  All  conflicts  have  been  handled  and  any  manual  resolutions  have 

occurred. 

•  All  instrumentation  has  been  configured  properly 

•  Competing  contractors  are  not  present 

Actors 

•  Instrumentation  Support  Personnel  (ISP) 

•  Server 

•  System  Under  Test  (SUT) 

•  Test  Engineer  (TE) 

•  Test  Harness 

Use  cases 
involved 

None 

Dependencies 

•  Test  is  on  authoritative  schedule 

•  Instrumentation  Configuration  Plan  is  complete  and  available 

•  Final  Test  Plan  is  complete  and  available 

Process  flow 

Step  description 

Artifact 

1 .  Instrumentation  is 
configured 

2.  Test  is  executed 

Raw  Data  is  generated 

3.  Raw  data  is  collected  and 
stored  on  the  server 

User  notified  of  location 

4.  TE  configures  security  on 
raw  data 

Properly  Secured  Raw  Data 

Deliverables 

•  Raw  Test  Data 

Additional 

infonnation 

None 

Table  6.  Use  Case  6:  Test  Execution 
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UC7 

Post-test  Data  Analysis 

Description 

Raw  data  is  reduced  by  the  Test  Engineer  (TE)  into  a  useable  form. 

Desired 

outcome 

While  reducing  the  raw  data  the  TE  begins  evaluation  of  the  System 

Under  Test  (SUT)  and  has  enough  information  to  generate  a  fonnal  Test 
Report  (TR). 

Assumptions 

None 

Actors 

•  Data  Processor 

•  Test  Engineer  (TE) 

•  Server 

Use  cases 
involved 

None 

Dependencies 

•  Raw  Test  Data 

Process  flow 

Step  description 

Artifact 

1 .  Retrieve  raw  data 

2.  Analyze  raw  data 

Reduced  Data 

3.  Store  reduced  data 

Deliverables 

•  Reduced  test  data 

Additional 

infonnation 

None 

Table  7.  Use  Case  7:  Post-test  Data  Analysis 


UC8 

Test  Report  Generation 

Description 

Compilation  of  a  Final  Test  Report  based  on  the  Test  Engineer’s  analysis 
of  the  reduced  data. 

Desired 

outcome 

A  fully  documented  Final  Test  Report  that  is  delivered  to  the  Program 
Manager  and  is  used  by  senior  leadership  to  evaluate  the  effectiveness  of 
the  System  Under  Test  (SUT). 

Assumptions 

None 

Actors 

•  Program  Manager  (PM) 

•  Reviewers 
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•  Server 

•  Test  Engineer  (TE) 

•  VDLS 

Use  cases 
involved 

•  Document  Collaboration  (UC9) 

Dependencies 

•  Reduced  Test  Data 

Process  flow 

Step  description 

Artifact 

1 .  Using  reduced  test  data 
create  draft  Test  Center 
(TC)  Test  Report  (TR) 

Draft  Test  Center  Test  Report 

2.  Begin  Draft  Approval 
Workflow  (UC9) 

3.  Final  TC  TR  Approved 

Final  Test  Center  Test  Report 

4.  Upload  Final  TC  TR  to 
VDLS 

VDLS  Populated 

5.  Notify  Test  Manager  (TM) 

User  Notified 

6.  Collect  all  final  TC  TRs 
from  VDLS 

7.  Create  draft  Final  TR 

Draft  Final  Test  Report 

8.  Begin  Draft  Approval 
Workflow  (Document 
Collaboration  UC9) 

9.  Final  TR  Approved 

Final  Test  Report 

10.  Notify  Program  Manager 
(PM) 

User  Notified 

Deliverables 

•  Draft  Test  Report 

•  Final  Test  Report 

•  VDLS  Populated 

Additional 

infonnation 

None 

Table  8.  Use  Case  8:  Test  Report  Generation 
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UC9 

Document  Collaboration 

Description 

Transformation  of  the  draft  Test  Report  into  a  Final  Test  Report  through 
an  approval  workflow  with  all  pertinent  parties. 

Desired 

outcome 

To  have  an  approved  Final  Test  Report  that  all  pertinent  parties  have 
been  able  to  review  prior  to  release. 

Assumptions 

None 

Actors 

•  Contributors 

•  Reviewers 

Use  cases 
involved 

None 

Dependencies 

•  Draft  Test  Report 

Process  flow 

Step  description 

Artifact 

1 .  Draft  Report  e-mailed  to 
reviewers 

2.  Draft  Report  Reviewed 

Comments 

3.  Draft  Report  Approved 

Final  Test  Report 

Deliverables 

•  Final  Test  Report 

Additional 

information 

Steps  1  and  2  are  continued  until  all  comments  are  addressed  and  all 
reviewers  approve  the  draft. 

Table  9.  Use  Case  9:  Document  Collaboration 
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IV.  CLOUD  BASED  T&E  PROCESS 


A.  INTRODUCTION 

Currently  within  ATEC,  a  place  does  not  exist  where  a  PM  can  go  and  obtain  all 
of  the  information  related  to  an  acquisition  program,  regardless  of  how  many  TCs  that 
program  crosses,  that  is  the  “one  stop  shop”,  or  integrated  working  environment  (IWE), 
does  not  exist.  Instead,  the  PM  must  rely  on  contacting  individuals  familiar  with  the 
program  to  obtain  the  information.  In  this  chapter,  we  explore  how  cloud  computing 
could  be  used  to  improve  upon  current  workflow  processes  utilized  by  PMs  within 
ATEC.  Currently  PMs  must  first  contact  the  TM,  who  then  contacts  the  TD,  who  then 
contacts  the  TE,  who  then  tells  the  TD  when  the  test  is  scheduled  for  range  time  (Figure 
32). 


:  Program  Manager  \  :  Test  Manager  :  Test  Director :  Resource  Manager  :  Financial  System  | 

- , - - - 1  L - , - - 1 - *  - , - -1  - f— - 1 


1 :  Request  (Test  Status) 


2:  Request  (Test  Status) 


,11:  Return  (Data) 


1 2:  Compile  (Test  Status) 


.  1 3:  Return  (Test  Status) 


S:  Request  (Expenditure^)  I 


6:  Return  (Data) 


4:  Search  (Data) 


5:  Return  (Data) 


7:  Request  (Test  Schedule) 


1 0:  Return  (Data) 


:  Test  Engineer :  Schedule 

- F - 1  — i — 


8:  Search  (Data) 
.  9:  Return  (Data) 


Figure  32.  Program  Management  Data  Call  Collaboration  As-Is 


This  information  is  then  relayed  back  to  the  requestor  in  a  serial  manner,  with 
additional  delay  introduced  whenever  someone  in  the  chain  of  communication  is 
unavailable.  Even  though  this  scheduling  information,  in  most  cases,  is  available  via 
scheduling  tools  at  the  TC  level,  the  PM  does  not  have  access  to  the  locally  stored 
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scheduling  data.  The  same  scenario  exists  if  the  PM  wants  to  know  the  current  financial 
status  for  the  test.  While  the  PM  can  see  the  high-level  information  within  SOMARDS, 
the  information  available  may  not  be  current  due  to  the  posting  cycles  at  the  TC.  In  other 
words  storing  the  data  locally,  that  is  on  the  workstation,  is  an  impediment  to  information 
sharing  within  ATEC  and  with  ATEC’s  stakeholders. 

Given  that  the  information  available  at  the  TE  level  is  the  most  up-to-date 
information  about  the  test,  that  data  should  be  available  to  all  parties  with  a  need  to  know. 
This  combined  with  ready  access  to  the  financial  information  and  other  programmatic 
information  would  provide  senior  leadership  with  the  situational  awareness  about  their 
acquisition  programs  and  those  across  the  entire  command.  Ideally,  the  framework  for  the 
IWE  would  be  created  when  the  RFTS  is  submitted.  From  that  moment  on,  all 
information  related  to  that  program  would  be  accessible  from  the  IWE,  with  all  of  the 
data  along  with  the  applications  that  operate  on  the  data  residing  in  the  ATEC  cloud. 
Figure  33  depicts  the  cloud-enabled  program  management  data-call  process. 


Figure  33.  Program  Management  Data  Call  Process  in  the  Cloud 
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All  this  data  would  not  have  to  physically  reside  within  the  cloud-based  IWE. 
Initially,  the  information  could  reside  in  legacy  systems  and  be  consumed  through  Web 
services  or  reside  in  a  cloud  APC  (Figure  34).  The  IWE  would  be  merely  a  front  end  to  a 
search-and-index  tool  with  interfaces  to  raw  unstructured  data  (e.g.,  cloud  APC  fde 
system,  existing  TC  data  center,  documents,  raw  test  data,  video,  images),  structured  data 
(e.g.,  relational  databases,  spreadsheet,  xml),  and  legacy  systems  (e.g.,  scheduling  tools, 
financial  tools). 


PM.  TM.  TD.  TE.  ISP 


Digital  Objects 
te  g  .  photos .  documents,  sj  l 


Figure  34.  Cloud  T&E  Architecture 


Does  this  solution  require  cloud  computing?  No,  it  could  be  done  today  through 
extremely  tight  coupling  of  existing  systems.  However,  that  rigid  coupling  is  difficult  to 
achieve  across  various  functional  divisions  within  one  TC  much  less  across  all  of  ATEC. 
This  has  been  one  of  the  stumbling  blocks  that  have  prevented  a  true  “soup-to-nuts” 
enterprise  solution  from  being  deployed.  Within  a  cloud  environment  though,  this  could 
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be  accomplished  with  less  rigidity  through  the  use  of  SaaS  and  the  migration  of  data  into 
the  cloud. 

B.  CLOUD  BASED  T&E  MISSION  THREAD 

The  Use  Cases  from  Chapter  III,  which  have  the  potential  to  change  the  most 
within  a  cloud  environment,  will  be  expanded  upon  in  the  following  sections  with  an 
overview  of  potential  new  functionality  described,  along  with  the  mission  thread 
described  within  Chapter  III.C. 

Whether  explicitly  mentioned  or  not,  at  every  level  of  the  processes  described 
below  role-based  access  controls  (RBAC)  are  needed  to  protect  sensitive  data.  People 
within  the  enterprise  can  be  assigned  to  roles  with  pennissions  associated  with  the  roles. 
The  details  for  applying  RBAC  in  the  cloud  are  being  investigated  within  the  cloud- 
security  communities.  It  is  worth  mentioning  that  at  the  TD  and  TE  level  there  will 
potentially  be  concerns  about  the  PM  having  the  level  of  detail  described  below:  such 
access  could  be  viewed  as  opening  the  door  to  micromanagement.  The  detailed  design  of 
such  security  access  controls  and  the  industrial  organization  psychology  and  management 
aspects  of  sharing  data  enterprise-wide  is  outside  the  scope  of  this  thesis. 

1.  Program  Management  in  the  Cloud 

While  moving  data  and  applications  to  the  cloud  would  provide  additional 
functionality  through  the  mash-up  of  previously  disconnected  datasets,  and  non- 
interoperable  applications  for  manipulating  the  data,  the  process  would  not  change 
dramatically.  The  process  would  still  begin  with  a  request  for  test  services  that  would 
automatically  start  a  workflow  to  identify  a  TM  and  create  a  cost  estimate.  Ah  of  the 
current  ties  to  ADSS  and  VDLS  would  either  still  exist  as-is  or  would  be  simplified 
through  the  replacement  of  VDLS  with  the  IWE  (Figure  35).  In  essence,  VDLS  and 
ADSS  would  be  just  another  data  source  for  the  IWE  to  pull  information  from. 


78 


Program  Manager 


Test  Director 


f 

I  Request  Test  Services  J- 


Create  ADSS  Shell  .. 


-j  Input  Requirements  | 

■1'  ' 

-I  Create  Program  Shell  | 


( Gather  Requirements  (•- 


i  Notify  | 


I  Activate  Effort  ^ — 


—I  Review  ) — 


h  Select  Test  Center  i 


Update  POC  j— 


I  Update  Milestones  k 


I  Gather  Requirements  |* 

- -J- - 

-]  Select  Lead  J _ 


1 

— (  Notify  | 


I  Gather  Requirements  t- 


Generate  Test  Plan 


— 4  Update  POC  j 

- 1  Notify  | 

— H  Store  j 


C  T‘g  ) 


: 


j 


— <  Notify  h- 


I  Notify 


{Review  I 
Test  Plan  | 

Select  Lead 


— H  Review  | 


Perform  Test 


I 


Figure  35.  Program  Management  Process  in  the  Cloud 


The  overall  collaborations  within  the  program  management  process  would  be 
essentially  the  same  as  the  current  system.  The  only  notable  difference  would  be  the 
replacement  of  VDLS  with  the  IWE  and  the  replacement  of  all  current  e-mail 
communications  with  collaboration  through  the  IWE  (Figure  36). 
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:  Program  Manager  ^  |  |  :  ADSS  j  |  :  Test  Manager  ^  |  |  :  Test  Center  |  [  :  Test  Director 


1 :  Request  Test  Services  (Requirements) 


:  IWE 

— ! - 


:  Supervisor  j  |  :  Test  Engineer  ^ 


3:  Create  ADSS  Effort  Shell 


J\  Activate  (Effort) 


5:  Gather  (Requirements) 


- 1- 

6;  Select  (Test  Center)  1 


I 


2:  Create  (Program 


Sh< 


11:  Update  (PQC) 


;r  (Requirements^) 


A  (Test  Director  j) 


12:  Update  fPOO 


17:  Update  (Milestones) 


I 

13:  Notify 


enerate  (Test  Plan) . 


r  (Requirement^ 


1 6:  Store  Tag  Secure 


19:  Notify 


Review  (Test  Pli 


^1:  Select  (Test  Engi  n 


22:  Notify  | 


23:  Review  fTest  ftlanl 


Figure  36.  Program  Management  Collaborations  in  the  Cloud 


Where  the  current  process  and  the  cloud  process  would  differ  the  most  is  within 
the  monitoring,  reporting,  and  control  aspect  of  program  management,  and  the  ability  of 
the  cloud  solution  to  provide  a  consistent  up-to-date  view  of  the  data  across  the 
enterprise.  The  current  method  for  monitoring,  reporting,  and  controlling  a  program  relies 
on  a  primarily  manual  process.  In  contrast,  a  cloud-based  system  could  permit  a  certain 
amount  of  automation  of  the  information  processing  and  sharing  functions,  essentially 
eliminating  the  middleman,  or  put  another  way,  the  human-in-the-loop  for  updating  the 
information  displayed  by  the  IWE,  as  shown  in  Figure  37. 
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:  Program  Manager 

:rWE  * 

:  Schedule 

Figure  37.  Program  Management  Data  Call  Collaboration  in  the  Cloud 

2.  Conduct  Test  Process 

Just  as  in  the  process  described  in  Chapter  III,  the  conduct  test  process  within  the 
cloud  revolves  heavily  around  the  TE  conducting  and  managing  the  day-to-day  status  for 
each  test.  Each  selected  TE  is  responsible  for  either  performing  or  coordinating  all 
aspects  of  their  portion  of  the  test  plan,  covering  everything  from  pretest  test  planning, 
test  execution,  post-test  analysis  and  data  verification,  to  collaboratively  writing  the  final 
test  report.  Within  a  cloud  environment,  the  main  items  that  will  change  will  be  the  data 
storage  location  and  the  test  report  process. 

a.  Pretest  Setup  Process 

Since  conducting  a  test  is  potentially  a  destructive  and  costly  process, 
coordination  is  needed  among  those  people  involved  in  planning  for  tests.  That  way  the 
test  does  not  have  to  be  repeated  at  a  later  date  due  to  poor  planning.  This  coordination 
occurs  during  the  pretest  planning  and  setup  process.  Currently,  the  pretest  planning  and 
setup  process  relies  heavily  on  informal  interactions  between  the  TE  and  other  TEs  in 
addition  to  range  personnel.  Most  of  these  interactions  occur  in  verbal  communications 
and  as  such  are  not  being  captured  in  a  manner  that  is  searchable  or  retrievable  at  a  later 
date.  When  personnel  leave  the  organization,  their  knowledge  of  the  programs  and 
workflow  processes  disappears  with  them.  However,  if  the  pretest  planning  and  setup 
process  took  advantage  of  cloud-based  social  media  tools  then  these  informal 
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collaborations  could  be  captured  where  they  could  be  tagged,  indexed,  searched,  and 
archived  (Figure  38).  Examples  of  such  tools  include  IBM’s  LotusLive,  Salesforce 
Chatter,  and  Facebook. 


Figure  38.  Pretest  Setup  Process 


Having  the  ability  to  search  through  these  interactions  would  help  DoD 
amass  the  large  amount  of  undocumented  corporate  knowledge  employees  currently 
posses  in  their  heads  into  documented  and  searchable  data.  The  IWE  could  provide  the 
nexus  for  the  social  media  tools  that  would  make  this  possible.  Tools,  such  as  online 
document  editors,  instant  messaging,  threaded  message  boards,  wikis,  blogs,  tags,  status 
updates,  ratings,  polls,  news,  hot  topics,  tasks,  RSS  feeds,  tweets,  and  so  on  would  all 
provide  for  a  rich  collaborative  environment. 
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The  TE  will  continue  to  work  with  the  ISP  to  identify  which  ground  truth 
data  elements  the  TE  will  need  to  evaluate  the  SUT.  The  ISP  will  use  that  infonnation  to 
determine  what  instrumentation  is  required  for  the  test,  and  where  it  should  be  placed  to 
capture  the  required  ground  truth  data  (Figure  39).  However,  within  a  cloud  environment 
this  collaboration  could  occur  within  the  tools  in  the  IWE,  which  would  provide  all  of  the 
benefits  that  were  previously  mentioned.  The  Instrumentation  Configuration  Plan  could 
be  created  and  tagged  within  the  IWE.  The  plan  could  theoretically  be  created  on  a 
smartphone  or  tablet  PC  and  uploaded  to  the  IWE  directly  from  the  field  through  for 
instance  a  wi-fi  or  3G  connection. 
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Figure  39.  Pretest  Setup  Collaborations 


b.  Schedule  Test  Process  in  the  Cloud 

The  process  and  collaborations  for  adding  an  event  to  the  schedule  and  for 
event  deconfliction  would  not  change  from  the  current  process.  However,  having  the 
schedule  information  accessible  through  the  cloud  would  allow  for  rapid  development  of 
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new  capabilities  for  the  PM  and  all  test  personnel.  One  such  function  would  be  the  ability 
for  a  PM  to  rapidly  answer  data  calls,  address  scheduling  conflicts  and  slippages,  and  so 
on. 

For  instance,  if  a  program  has  multiple  phases  that  cross  TCs  then  this 
capability  would  give  the  PM  visibility  into  TC  Alpha,  Beta,  Charlie,  and  Delta’s 
schedules.  Within  the  IWE,  the  PM  could  establish  true  dependencies  between  the  test 
milestones,  regardless  of  what  TC  the  testing  will  occur  at.  The  test  personnel  would  also 
benefit  from  the  additional  visibility  into  predecessor’s  schedules  that  the  IWE  would 
provide.  If  Alpha’s  schedule  slips  and  Beta’s  start  date  is  dependent  on  Alpha’s  test 
completing,  then  the  POC  at  Beta  could  be  automatically  notified  that  Alpha  has  slipped, 
so  Beta  could  adjust  their  schedule  as  necessary. 


3,  Test  Execution  Process  in  the  Cloud 

The  test  execution  process  is  essentially  the  same  in  the  cloud  with  the  only 
changes  being  where  the  data  is  stored  (Figure  40). 
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Figure  40.  Test  Execution  Process  in  the  Cloud 
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The  cloud-enabled  collaborations  involved  in  the  test  execution  process  are  also 
similar  to  the  existing  one.  However,  with  the  cloud  approach,  access  control  can  be 
enforced  in  a  unifonn  manner  by  the  cloud  services  across  the  enterprise  (Figure  41). 
Where  the  data  is  saved  and  how  the  security  is  set  will  be  expanded  on  more  in  the 
following  sections. 
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Figure  41.  Test  Execution  Collaborations  in  the  Cloud 


4.  Post-test  Data  Analysis  Process  in  the  Cloud 

For  a  cloud  environment  to  be  a  viable  candidate  for  storing  and  processing  T&E 
level  1  data  (Table  10),  it  must  have  sufficient  storage  capacity,  support  efficient  search 
relying  on  standard  or  custom  tagging  (similar  to  G-mail™),  enforce  access  controls, 
support  legacy  data-reduction  tools,  save  the  reduced  data  back  to  the  cloud  with  possibly 
new  tagging,  and  provide  non-reputable  audit  trails  of  access  to  the  data  and  applications. 
The  cloud  must  allow  for  the  storage  of  structured  and  unstructured  data,  with  and 
without  associated  metadata,  and  allow  users  to  associate  custom  tags  to  data. 
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Data  Level 

Description 

Possible  Sources 

Examples  of  Content 

Disposition 

Level  1 
“raw  data” 

Data  in  their 
original  form. 
Results  of  field 
trials  just  as 
recorded. 

Complete  data 
collection  sheets, 
exposed  camera  film, 
voice  recording  tapes, 
original 

instrumentation 
magnetic  tape  or 
printouts,  original 
videotapes,  filled 
questionnaires, 
interview  notes 

1 .  All  reported  target 
presentations  and  detection 

2.  Clock  times  of  all  events 

3.  Azimuth  and  vertical 
angle  from  each  flash  base 
for  each  flash 

4.  Recording  tapes  of 
interviews 

Accumulated 
during  trials  for 
processing. 
Usually 
discarded  after 
use.  Not 
published 

Level  2 
“reduced 
data” 

Data  taken  from 
the  raw  form  and 
consolidated. 
Invalid  or 
unnecessary  data 
points  identified 
as  such  with 
supporting 
rationale. 

Confirmed  and 
corrected  data 
collection  sheets,  film 
with  extraneous 
footage  identified 
corrected  tapes  or 
printouts,  and  original 
raw  data  with  “No 
test”  events  identified. 

1.  Record  of  all  valid 
detections. 

2.  Start  and  stop  times  of  all 
applicable  events. 

3.  Computed  impact  points 
off  each  round  flashed. 

4.  Confirmed  interview 
records. 

Produced 
during 
processing. 
Usually  files 
after  use.  Not 
published 

Level  3 
“ordered 
data” 

Data  that  have 
been  checked  for 
accuracy  and 
arranged  in 
convenient  order 
for  handling. 
Operations 
limited  to 
counting  and 
elementary 
arithmetic. 

Spreadsheet,  tables, 
typed  lists,  ordered 
and  labeled  printouts, 
purified  and  ordered 
tape,  edited  film, 
edited  magnetic  tapes. 

1 .  Counts  of  detections 
arranged  in  sets  showing 
conditions  under  which 
detections  occurred. 

2.  Elapsed  times  by  type  of 
event. 

3.  Impact  points  of  rounds 
by  condition  under  which 
fired. 

4.  Interview  comments 
categorized  by  type. 

Not  usually 
published  but 
made  available 
to  analysts. 
Usually  stored 
in  institutional 
data  banks.  All 
or  part  may  be 
published  as 
supplements  to 
the  test  report 

Level  4 
“findings” 
or  summary 
statistics 

Data  that  have 
been 

summarized  by 

elementary 

mathematical 

operations. 

Operations 

limited  to 

descriptive 

summaries 

without 

judgments  or 

inferences.  Does 

not  go  beyond 

what  was 

observed  in  the 

test. 

Tables  or  graphs 
showing  totals,  means, 
medians,  modes, 
maximums, 
minimums,  quartiles, 
deciles,  percentiles, 
curves,  or  standard 
deviations.  Qualitative 
data  in  form  of  lists, 
histographs,  counts  by 
type,  or  summary 
statements. 

1.  Percentage  of 
presentations  detected. 

2.  Mean  elapsed  times. 

3.  Calculated  probable 
errors  about  the  centers  of 
impact. 

4.  Bar  graph  showing 
relative  frequency  of  each 
category  of  comment. 

Published  as  the 
basic  factual 
findings  of  the 
test. 

Level  5 
“analysis” 
or 

Data  resulting 
from  statistical 
tests  of 

Results  of  primary 
statistical  techniques 
such  as  T-tests,  Chi- 

1 .  Inferred  probability  of 
detection  with  its 
confidence  interval. 

Published  in 

evaluation 

reports. 
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inferential 

statistics 

hypothesis  or 
interval 
estimation. 
Execution  of 
planned  analysis 
data  Includes 
both 

comparisons  and 
statistical 
significance 
level.  Judgments 
limited  to 
analyst’s 
selection  of 
techniques  and 
significant 
levels. 

square,  F-test,  analysis 
of  variance,  regression 
analysis,  contingency 
table  analyses  and 
other  associated 
confidence  levels. 
Follow-on  tests  of 
hypotheses  arising 
from  results  of  earlier 
analysis,  or  fallback  to 
alternate  non- 
parametric  technique 
when  distribution  of 
data  does  not  support 
assumption  of 
normality.  Qualitative 
data  in  the  form  of 
prevailing  consensus. 

2.  Significance  of  difference 
between  two  mean  elapsed 
times. 

3.  Significance  of  difference 
between  observed  probable 
error  and  criterion  threshold. 

4.  Magnitude  of  difference 
between  categories  of 
comments. 

(If  evaluation 
report  is  part  of 
test  report,  the 
level  5  analysis 
results  are 
presented 
separately  from 
the  level  4 
findings.) 

Level  6 
“extended 
analysis”  or 
operations 

Data  resulting 
from  further 
analytic 

treatment  going 
beyond  primary 
statistical 
analysis, 
combination  of 
analytic  results 
from  different 
sources,  or 
exercise  of 
simulation  or 
models. 

Judgments 
limited  to 
analysts’  choices 
only. 

Insertion  of  test  data 
into  a  computational 
model  or  a  combat 
simulation, 
aggregation  of  data 
from  different  sources 
observing  required 
disciplines,  curve 
fitting  and  other 
analytic 

generalization,  or 
other  operations 
research  techniques 
such  as  application  of 
queuing  theory, 
inventory  theory,  cost 
analysis,  or  decision 
analysis  techniques. 

1 .  Computation  of 
probability  of  hit  based  on 
target  detection  data  from 
test  combined  with  separate 
data  or  probability  of  hit 
given  detection. 

2.  Exercise  of  attrition 
model  using  empirical  test 
times  distribution. 

3.  Determination  of  whether 
a  trend  can  be  identified 
from  correlation  of  flash 
base  accuracy  data  under 
stated  conditions  from 
different  sources. 

4.  Delphi  technique 
treatment  of  consensus  of 
interview  comments. 

Published  as 
appropriate  in 
evaluation 
reports. 

Level  7 
“conclusion 
”  or 

evaluation 

Data  conclusions 
resulting  from 
applying 
evaluative 
military 
judgments  to 
analytic  results. 

Stated  conclusions  as 
to  issues,  position 
statements,  challenges 
to  validity  or  analysis. 

1 .  Conclusion  as  to  whether 
probability  of  detection  is 
adequate. 

2.  Conclusion  as  to 
timeliness  of  system 
performance. 

3.  Conclusion  as  to  military 
value  of  flash  base 
accuracy. 

4.  Conclusion  as  to  main 
problems  identified  by 
interviewees. 

Published  as  the 
basic  evaluative 
conclusions  of 
evaluation 
reports. 

Table  10.  Levels  of  Data  from  (ATEC,  2004) 
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Removing  the  data  processing  from  the  local  desktop  and  placing  it  in  a  scalable 
cloud  environment  would  address  the  data  reduction  performance  requirements,  provided 
the  bandwidth  between  the  user  and  the  reduction  tools  is  adequate.  This  could  reduce  the 
amount  of  time  required  for  data  reduction  while  also  providing  additional  capabilities  for 
users.  Moving  the  data  and  tools  to  the  cloud  could  also  help  eliminate  or  at  least 
minimize  the  number  of  copies  of  data  that  are  currently  stored. 

In  a  cloud  solution,  there  would  be  minor  changes  in  the  Post-test  Data  Reduction 
process.  Such  as  the  location  where  the  data  is  saved  post  test,  where  the  reduction  tools 
reside,  and  the  location  where  the  TE  perfonns  the  reduction.  Rather  than  the  ISP  storing 
the  collected  data  on  a  local  range  server  it  could  be  stored  in  a  cloud  data  center. 

Going  back  to  the  scenario  described  in  Chapter  III,  the  ISP  know  which 
scheduled  test  the  data  is  being  collected  for.  So,  with  the  creation  of  an  IWE  interface, 
the  scheduling  meta-data  could  automatically  be  associated  with  all  data  uploaded  to  the 
data  center  by  the  ISP  by  selecting  from  a  list  of  test  events  that  were  completed  at  their 
range  on  a  certain  date.  At  the  time  of  upload,  the  ISP  could  also  tag  the  uploaded  data 
with  custom  metadata  that  they  think  is  pertinent.  The  metadata  will  provide  a 
mechanism  for  the  indexing  and  searching  of  the  raw  data  at  a  later  date. 

When  the  metadata  from  the  schedule  is  pulled  into  the  system,  the  initial 
pennissions  could  also  be  established  to  restrict  access  to  only  the  POC  that  was  listed  in 
the  schedule  tool,  who  is  likely  the  TE  (Figure  42).  That  TE  could  then  access  the  IWE 
from  any  approved  computing  device  with  network  connectivity  to  the  cloud,  viewing  the 
latest  data  uploaded  by  the  ISP. 
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Figure  42.  Post- test  Data  Analysis  Process  in  the  Cloud 


The  TE  would  have  the  ability  to  set  permissions  on  the  data,  add  additional 
custom  tags,  or  begin  the  process  of  analysis  and  reduction  (Figure  43)  using  tools  based 
in  the  cloud.  During  analysis  and  reduction,  the  TE  and  needed  tools  would  merely 
reference  the  data  located  in  the  cloud  data  center  rather  than  the  local  range  server. 
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Figure  43.  Post-text  Data  Analysis  Collaborations  in  the  Cloud 


Duplicate  data  reduction  could  be  obtained  through  reduction  of  logical  copies, 
however,  not  physical  copies.  There  is  an  important  but  subtle  difference  between  the 
two.  Physical  copies  are  made  by  organizations  for  either  fault  tolerance  or  performance 
reasons  and  the  enterprise  system  knows  how  to  handle  these  files  (e.g.,  file  backup  from 
June  1,  2010).  A  logical  copy  refers  to  copies  of  the  same  file  or  information  at  multiple 
locations  that  are  not  related,  which  the  enterprise  knows  nothing  about,  only  the  user 
knows  that  they  are  related.  For  example,  while  reducing  level  1  data  a  TE  may  end  up 
with  four  copies  of  a  single  file:  one  on  the  TC  enterprise  shared  projects  server  location 
(for  sharing  with  others),  one  on  a  notebook  hard  drive  (for  access  while  TDY),  one  on  a 
desktop  hard  drive  (for  heavy  processing  while  in  the  office),  and  one  on  a  private  storage 
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location  on  the  server  (user’s  manual  file  versioning/backup  solution)  (Foster  et  ah, 
2010a).  The  relationship  between  all  of  these  versions  is  only  known  to  the  TE. 

To  make  the  matter  worse,  any  of  the  logical  copies  that  are  stored  on  an 
enterprise  server  also  have  physical  copies.  By  moving  the  data  and  reduction  tools  into 
the  cloud,  four  logical  copies  for  one  user  have  just  been  turned  into  one  copy.  Having  a 
single  copy  of  the  data  would  also  reduce  the  configuration  management  and  control 
issues  that  arise  from  multiple  users  working  on  the  same  document  and  creating  multiple 
logical  copies.  Moving  the  data  to  the  cloud  also  provides  users  the  option  to  access  the 
data  through  the  I  WE  from  anywhere  and  via  any  authorized  device.  This  capability 
could  eliminate  the  need  for  logical  copies  on  both  the  desktop  and  notebook  hard  drives. 

a.  Test  Report  Generation  Process  in  the  Cloud 

The  test  report  generation  process  within  the  cloud  (Figure  44)  would  be 
much  simpler  than  the  current  process  with  all  actors  being  abstracted  into  three  generic 
roles:  Author,  Reviewer,  and  Customer.  In  the  current  process  multiple  actors  fill  the  role 
of  Author  and  Reviewer  depending  on  where  they  were  in  the  process.  For  example,  the 
TE  and  TM  are  both  authors,  the  TE  of  the  draft  TE  TR  and  the  TM  for  the  compilation 
of  the  final  TR,  and  since  the  TE  delivers  its  report  to  the  TM  then  the  TM  can  also  be 
considered  a  customer. 
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Figure  44.  Test  Report  Generation  Process  in  the  Cloud 


The  test  report  generation  collaborations  in  the  cloud  process  would 
revolve  around  the  IWE  (Figure  45).  Since  everything  would  be  accessible  from  within 
the  IWE,  all  actors  would  be  able  to  perform  their  collaborations  within  the  same 
environment.  The  passing  of  artifacts  back  and  forth  through  e-mail, 
uploading/downloading  of  TRs  to/from  a  middleman  storage  area,  and  the  manual  e-mail 
notifications  would  be  gone.  The  manual  notifications  could  be  replaced  with  a  message 
board,  wiki,  etc.  within  the  IWE  or  simply  with  automated  notifications.  More  detailed 
information  about  the  collaborations  within  the  IWE  is  provided  in  the  next  section. 
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Figure  45.  Test  Report  Generation  Collaborations  in  the  Cloud 


In  addition  to  streamlining  the  current  process,  having  all  of  the  TR 
information  within  the  cloud  would  provide  new  capabilities.  Having  the  various  TR 
workflows  accessible  by  the  IWE,  combined  with  the  previously  mentioned  scheduling 
information  would  provide  a  new  capability  for  all  actors.  The  new  capability  would 
allow  all  actors  post  test  to  track  where  the  TR  is  at  in  the  generation  process. 

Having  this  information  readily  available  would  make  it  easier  to  establish 

standard  metrics  for  time  taken  from  the  end  of  a  test  event  through  the  delivery  of  a  final 

TR.  This  information  could  be  visible  to  all  involved  actors  (TE,  Supervisor,  TD,  TM, 
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and  PM)  with  everyone  able  to  see  how  long  the  TR  stays  in  a  particular  location  before 
moving  on  to  the  next  reviewer  or  contributor.  In  other  words,  anyone  with  the  proper 
access  rights  could  view  where  the  TR  is  at  in  the  overall  process,  and  ultimately,  any 
bottlenecks  in  the  process  would  be  readily  identified  and  resolved.  As  authors  and 
reviewers  made  updates  to  the  TR,  they  could  add  notes  or  tags  into  the  status  to  indicate 
potential  delays.  That  information  could  then  be  rolled-up  into  an  overall  status  for  the 
PM,  who  in  turn  could  drill  down  to  see  that,  for  example,  the  TE  report  at  TC  Alpha  is 
delayed  at  functional  Division  Delta  while  the  TC  Beta  final  TR  is  in  the  final  steps  of 
review. 

These  TR  delivery  metrics  could  be  one  tool  available  for  senior 
leadership  to  use  in  determining  if  moving  to  a  cloud-based  environment  increased 
efficiency.  Through  the  compilation  of  an  average  TR  delivery  time  from  the  completion 
of  a  test  until  delivery  of  the  final  TR,  over  many  different  programs,  for  a  period  of  time 
before  and  after  a  move  to  a  cloud  environment  senior  leadership  would  have  a  good 
representation  of  the  effect  of  moving  to  a  cloud. 

5.  Document  Collaboration  Process  in  the  Cloud 

Currently,  most  of  the  interactions  for  reviewing  and  collaborating  on  documents 
occur  through  e-mail.  This  process  could  be  greatly  improved  through  the  usage  of  social 
networking  tools  accessed  through  the  IWE.  Doing  so  would  allow  anyone  with  access  to 
quickly  see  what  has  occurred  recently  with  the  program.  So,  if  a  TE  goes  TDY  during 
the  middle  of  collaborating  on  the  final  TR,  then  while  on  TDY,  the  TE  could  view  what 
activities  were  occurring,  what  communications  were  ongoing  between  the  various 
editors,  status  of  scheduling,  and  continue  to  stay  aware  of  the  current  situation  all  while 
on  TDY. 

If  an  online  document  editor  were  utilized  for  the  creation  and  editing  of  the  TR, 
then  the  TE  could  continue  to  collaborate  and  work  on  the  document  regardless  of  where 
the  TE  was  physically  located.  The  author,  contributors,  reviewers,  and  customer  would 
all  be  users  of  the  same  online  document  editing  tool.  However,  each  would  have 
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different  permissions  and  would  be  participants  at  different  stages  in  an  overarching 
workflow.  A  change  in  status  of  some  arbitrary  metadata  (such  as  changing  from  draft  to 
final)  could  signal  the  document  editing  tool  to  start  a  workflow.  The  workflow  could 
send  notification  to  the  reviewers  that  the  document  is  ready  for  review. 

This  notification  could  be  sent  by  various  means  such  as  e-mail,  tweet,  hot  topic 
posting,  RSS  feed,  etc.  The  reviewers  would  then  go  to  the  IWE,  review  the  document, 
and  either  approve  the  document,  which  would  change  the  metadata  from  draft  to  final 
resulting  in  the  next  workflow  participant  being  notified.  Or  the  reviewer  could  suggest 
changes  to  the  document  by  adding  comments  either  into  the  document  or  into  a  message 
board  within  the  program  workspace.  The  contributor  that  started  the  review  workflow 
step  would  then  be  notified  and  the  process  would  start  over  (Figure  46). 


Figure  46.  Document  Collaboration  Process  in  the  Cloud 
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c. 


USE  CASE  ANALYSIS  SUMMARY 


The  Use  Case  diagram,  shown  below  (Figure  47),  is  unchanged  from  Chapter  III 
and  is  displayed  only  for  the  convenience  of  the  reader.  Just  as  in  Chapter  III,  the  model 
is  made  up  of  nine  use  cases,  described  in  Tables  1 1-19,  with  the  differences  between  the 
Chapter  III  scenarios  and  the  cloud  computing  scenarios  being  shown  in  italic.  A  table  is 
also  provided  to  summarize  all  T&E  processes  which  would  be  modified  in  a  move  to  a 
cloud-based  environment. 
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UC10 

Program  Management 

Description 

A  Program  Manager  (PM)  requests  a  new  LFT&E  test. 

Desired 

outcome 

A  test  program  is  initiated  with  the  test  requirements  flowing  from  the 

PM  to  the  appropriate  TC  and  eventually  to  the  appropriate  TE  for  the 
test. 

Assumptions 

None 

Actors 

•  Program  Manager  (PM) 

•  Supervisor 

•  System  Under  Test  (SUT) 

•  Test  Center  (TC) 

•  Test  Director  (TD) 

•  Test  Engineer  (TE) 

•  Test  Manager  (TM) 

Dependencies 

None 

Process  flow 

Step  description 

Artifact 

1.  PM  submits  a  Request  for  Test 
Services  (RFTS)  to  the  IWE 

RFTS  created 

2.  PM  Inputs  requirements  into 

Program  Shell  created 

IWE 

ADSS  Effort  Shell  created 

3.  TM  Notified 

4.  TM  Reviews  Requirements 

5.  TM  selects  Test  Center  (TC) 

IWE  TC  Updated 

6.  TM  activates  Effort 

Effort  activated  within  ADSS 

7.  TC  Notified 

8.  TC  Reviews  Requirements 

9.  TC  selects  TD 

IWE  POC  updated 

ADSS  TC  POC  field  updated 

10.  TD  Notified 

11.  TD  reviews  requirements 

12.  TD  generates  Test  Plan 

Initial  Test  Plan  created 

ADSS  Milestones  updated 
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VDLS  Folder  Structure  created 

Metadata 

13.  TD  selects  Division 

Metadata 

14.  Division  Supervisor  notified 

15.  Division  Supervisor  reviews 
requirements 

16.  Division  Supervisor  selects  TE 

Metadata 

17.  TE  Notified 

18.  TE  Reviews  test  plan 

19.  TE  performs  test 

Deliverables 

•  ADSS  Effort 

•  Initial  Test  Plan 

•  Project  Plan 

•  Test  Requirements 

Additional 

infonnation 

None 

Table  11.  Use  Case  10:  Program  Management 


UC11 

Conduct  Test 

Description 

Using  the  requirements  and  initial  test  plan  a  test  will  be  conducted.  This 
will  require  pretest  planning,  scheduling  coordination,  event 
deconfliction,  test  execution,  and  post  test  data  analysis. 

Desired 

outcome 

To  successfully  conduct  the  test,  obtain  the  data  elements  that  will  allow 
the  TE  to  properly  evaluate  the  SUT,  and  reduce  the  data  in  preparation 
for  the  generation  of  a  fonnal  Test  Report. 

Assumptions 

•  Funding  has  arrived,  safety  fans  have  been  created,  and 

environmental  impact  concerns  have  been  addressed. 

•  No  external  issues  exist  to  prevent  test  from  occurring. 

•  System  Under  Test  (SUT)  is  onsite  at  the  Test  Center  (TC) 

Actors 

•  Instrumentation  Support  Personnel  (ISP) 

•  Test  Engineer  (TE) 

•  System  Under  Test  (SUT) 
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Use  cases 
involved 

•  Post-text  Data  Analysis  UC 1 6 

•  Pretest  Setup  UC  12 

•  Test  Execution  UC  1 5 

Dependencies 

•  Initial  Test  Plan 

•  Test  Requirements 

Process  flow 

Step  description 

Artifact 

1 .  Pretest  setup 

Finalized  test  plan 

2.  Schedule  test 

Test  on  authoritative  schedule 

3.  Test  execution 

Raw  test  data  collected 

4.  Post  test  data  analysis 

Test  data  reduced 

Deliverables 

•  Finalized  Test  Plan 

•  Scheduled  Test 

•  Raw  Test  Data 

•  Reduced  Test  Data 

Additional 

infonnation 

None 

Table  12.  Use  Case  11:  Conduct  Test 


UC12 

Pretest  Setup 

Description 

Prior  to  test  execution  a  great  deal  of  upfront  planning  must  occur.  The 
test  plan  must  be  finalized,  instrumentation  must  be  configured, 
schedules  must  be  coordinated,  the  test  must  be  added  to  the  authoritative 
schedule,  and  test  setup  information  must  be  fully  documented. 

Desired 

outcome 

To  perform  the  proper  upfront  planning  and  coordination  for  the  test 
execution  to  be  repeatable  and  be  executed  with  as  few  oversights  as 
possible. 

Assumptions 

•  Test  Plan  has  been  finalized. 

Actors 

•  Instrumentation  Support  Personnel  (ISP) 

•  Schedule  Test  Process 

•  Scheduler 

•  Test  Engineer  (TE) 
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Use  cases 
involved 

•  Schedule  Test  Process  UC13 

Dependencies 

•  Initial  Test  Plan 

•  Test  Requirements 

Process  flow 

Step  description 

Artifact 

1 .  Test  Engineer  requests 
range  time* 

Schedule  Request 

2.  Test  scheduled  by  Schedule 
Test  Process*  (UC13) 

Authoritative  Schedule  for  SUT 

3.  Notify  ISP  and  TE 

4.  ISP  configures 
instrumentation 

Instrumentation  Configuration  Plan 

5.  Upload  Configuration  Plan 
to  IWE 

6.  Tag,  secure,  and  store  data 

Configuration  Plan  Meta  Data 

Deliverables 

•  Schedule  Request 

•  Instrumentation  Configuration  Plan 

•  Authoritative  Schedule  for  SUT 

Additional 

infonnation 

*  Step  1  and  2  will  continue  will  continue  until  the  requested  schedule 
successfully  goes  through  the  Schedule  Test  Process  (UC13). 

Table  13.  Use  Case  12:  Pretest  Setup 


UC13 

Schedule  Test 

Description 

Put  a  test  on  the  authoritative  range  schedule. 

Desired  outcome 

Successfully  schedule  range  time  and  all  required  resources  needed  to 
support  a  test. 

Assumptions 

•  All  required  resources  are  known  and  identified  prior  to  scheduling 
test 

Actors 

•  Event  De-confliction  Process 

•  Scheduler 

•  Test  Engineer 

Use  cases 
involved 

•  Event  De-confliction  (UC 14) 
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Dependencies 

•  Final  Test  Plan 

Process  flow 

Step  description 

Artifact 

1 .  *  Schedule  Request  received 
from  Test  Engineer 

Proposed  Request 

2.  *  Conflicts  identified  by 

Event  De-confliction  process 
(UC14) 

3.  Conflicts  resolved 

Test  Engineer  notified 

Test  added  to  Authoritative  Schedule 

Deliverables 

•  Authoritative  Schedule  for  SUT 

Additional 

information 

*  Step  1  and  2  will  continue  will  continue  until  all  conflicts  are  addressed. 

Table  14.  Use  Case  13:  Schedule  Test 


UC14 

Event  De-confliction 

Description 

Identify  and  resolve  potential  scheduling  conflicts  prior  to  test  events 
being  added  to  the  authoritative  schedule. 

Desired 

outcome 

To  have  a  fully  deconflicted  schedule,  so  there  is  not  a  schedule  delay 
due  to  lack  of  prior  coordination  between  test  programs. 

Assumptions 

None 

Actors 

•  Authoritative  Schedule  Information 

•  Conflictor 

•  Conflictor  Data 

•  Scheduler 

•  Test  Engineer 

Use  cases 
involved 

•  Schedule  Test  (UC13) 

Dependencies 

•  Proposed  schedule 

Process  flow 

Step  description 

Artifact 

1 .  Receive  proposed  schedule 

2.  Request  scheduled  events 

3.  Identify  conflicts 

List  of  conflicts 
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4.  Send  conflict  list  to  user 

User  Notified 

Deliverables 

•  List  of  conflicts 

Additional 

infonnation 

If  no  conflicts  are  present  then  the  user  is  notified  that  no  conflicts  were 
found.  This  process  will  repeat  with  UC13  until  all  conflicts  are 
addressed. 

Table  15.  Use  Case  14:  Event  De-confliction 


UC15 

Test  Execution 

Description 

LFT&E  test  occurs  and  data  is  collected. 

Desired 

outcome 

The  test  plan  is  executed  with  no  problems  with  data  being  successfully 
collected,  stored,  tagged,  and  secured  for  later  reduction  by  the  Test 
Engineer  (TE). 

Assumptions 

•  All  conflicts  have  been  handled  and  any  manual  resolutions  have 

occurred. 

•  All  instrumentation  has  been  configured  properly 

•  Competing  contractors  are  not  present 

Actors 

•  Instrumentation  Support  Personnel  (ISP) 

•  Data  Center 

•  Integrated  Working  Environment  (IWE) 

•  System  Under  Test  (SUT) 

•  Test  Engineer  (TE) 

Use  cases 
involved 

None 

Dependencies 

•  Test  is  on  authoritative  schedule 

•  Instrumentation  Configuration  Plan  is  complete  and  available  on 

IWE 

•  Final  Test  Plan  is  complete  and  available  on  IWE 

Process  flow 

Step  description 

Artifact 

1 .  Instrumentation  is 
configured 

2.  Test  is  executed 

Raw  Data  is  generated 

3.  Raw  data  is  collected  and 

POC  for  test  event  in  Schedule 
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uploaded  to  the  IWE 

Tool  is  notified  of  location 

4.  Data  is  tagged  and  secured 

Metadata 

Secured  Raw  Data 

Deliverables 

•  Raw  Test  Data 

Additional 

None 

infonnation 

Table  16.  Use  Case  15:  Test  Execution 


UC16 

Post-test  Data  Analysis 

Description 

Raw  data  is  reduced  by  the  Test  Engineer  (TE)  into  a  useable  form. 

Desired 

outcome 

While  reducing  the  raw  data  the  TE  begins  evaluation  of  the  System 

Under  Test  (SUT)  and  has  enough  information  to  generate  a  fonnal  Test 
Report  (TR). 

Assumptions 

None 

Actors 

•  Data  Center 

•  Test  Engineer  (TE) 

•  Integrated  Working  Environment  (IWE) 

Use  cases 
involved 

None 

Dependencies 

•  Raw  Test  Data 

Process  flow 

Step  description 

Artifact 

1.  Search  for  data 

2.  Retrieve  raw  data 

3.  Analyze  raw  data 

Reduced  Data 

4.  Tag,  Secure  reduced  data 

Metadata 

Secured  Reduced  Data 

Deliverables 

•  Reduced  test  data 

Additional 

infonnation 

None 

Table  17.  Use  Case  16:  Post-test  Data  Analysis 
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UC17 

Test  Report  Generation 

Description 

Compilation  of  a  Final  Test  Report  based  on  the  Test  Engineer’s  analysis 
of  the  reduced  data. 

Desired 

outcome 

A  fully  documented  Final  Test  Report  that  is  delivered  to  the  Program 
Manager  and  is  used  by  senior  leadership  to  evaluate  the  effectiveness  of 
the  System  Under  Test  (SUT). 

Assumptions 

None 

Actors 

•  Author 

•  Customer 

•  Integrated  Working  Environment  (IWE) 

•  Data  Center 

•  Reviewer 

Use  cases 
involved 

•  Document  Collaboration  (UC 18) 

Dependencies 

•  Reduced  Test  Data 

Process  flow 

Step  description 

Artifact 

1.  Search  for  reduced  test 
data 

2.  Create  Draft  Test  Report 

Draft  Test  Report 

3.  Tag,  Secure  Draft 

Metadata 

Secured  Draft 

4.  Begin  Draft  Approval 
Workflow  (UC18) 

5.  Notify  Reviewers 

6.  Search  for  Draft  Test 

Report 

7.  Review 

Final  Test  Report 

8.  Tag,  Secure  Final  Test 
Report 

Metadata 

Secured  Final  Test  Report 

9.  Notify  Program  Manager 
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Deliverables 

•  Draft  Test  Report 

•  Final  Test  Report 

Additional 

infonnation 

None 

Table  18.  Use  Case  17:  Test  Report  Generation 


UC18 

Document  Collaboration 

Description 

Transformation  of  the  draft  Test  Report  into  a  Final  Test  Report  through 
an  approval  workflow  with  all  pertinent  parties. 

Desired 

outcome 

To  have  an  approved  Final  Test  Report  that  all  pertinent  parties  have 
been  able  to  review  prior  to  release. 

Assumptions 

None 

Actors 

•  Contributors 

•  Reviewers 

•  Integrated  Working  Environment  (IWE) 

Use  cases 
involved 

None 

Dependencies 

•  Draft  Test  Report 

Process  flow 

Step  description 

Artifact 

1.  *  Notification  sent  to 

Reviewers 

2.  *  Search  for  Draft  Test 
Report 

3.  *  Review  Draft  Report 

Comments 

4.  Approve  Draft  Report 

Final  Test  Report 

5.  Tag  Report 

Metadata 

Deliverables 

•  Final  Test  Report 

Additional 

infonnation 

•  Steps  1,  2,  and  3  are  continued  until  all  comments  are 
addressed  and  all  reviewers  approve  the  draft. 

Table  19.  Use  Case  18:  Document  Collaboration 
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Capability 

Description  of  Change  due  to  Cloud  Environment 

Program  management 

Semi-automated  cloud-based  communications/collaborations 
within  the  IWE  could  replace  the  current  manual  serial  process 
for  the  monitoring,  reporting,  and  controlling  of  a  program. 

Cloud-based  data  storage 

Storing  data  locally  impedes  information  sharing  with  ATEC 
and  ATEC's  stakeholders.  Having  all  artifacts  either  stored  in 
the  cloud,  or  accessible  from  the  IWE  would  expedite  the 
sharing  of  data  amongst  all  stakeholders,  while  also  reducing  the 
number  of  copies  of  data  stored. 

Accessible  from 
anywhere 

Artifacts  within  the  IWE,  to  include  communications  and 
collaborations,  could  be  viewed  from  any  authorized  network 
enabled  device  (e.g.,  computer,  smartphone).  Legacy  data  and 
applications  could  be  accessed  through  the  IWE  until  the  data  or 
application  could  be  migrated  to  the  cloud. 

Collaboration 

Using  the  IWE  and  the  associated  cloud-based  collaboration  as 
the  primary  mechanism  for  collaboration  between  all  personnel 
involved  in  the  T&E  process  would  also  help  DoD  amass  the 
large  amount  of  undocumented  corporate  knowledge  employees 
currently  posses,  in  their  heads,  into  documented  and  searchable 
data. 

Data  tagging 

IWE  would  allow  for  automatically  tagging  and  securing  of  data 
based  on  available  meta-data  and  will  also  allow  users  to  add  or 
associate  custom  tags  to  data  as  needed. 

Data  reduction 

Data  reduction  tools  could  be  hosted  within  the  cloud  and 
accessed  through  the  IWE.  Data  reduction  could  occur  within 
the  IWE  with  reduced  artifacts  being  saved  back  to  the  IWE 
with  new  tags. 

Document  collaboration 

Document  reviews  could  occur  directly  within  the  IWE,  the 
document  could  start  out  within  the  IWE,  go  through  all  review 
processes  within  the  IWE  replacing  the  current  method  of 
passing  artifacts  back  and  forth  through  email  and  using 
middleman  storage  areas. 

Security 

All  data  accessed  via  the  IWE  would  be  protected  through  the 
use  of  RBACs  and  the  IWE  would  provide  non-reputable  audit 
trails  of  access  to  the  data  and  applications. 

Situational  Awareness 

Schedule  and  financial  information,  as  well  as  other 
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information,  could  be  viewed  across  all  test  centers  and  through 
a  mash-up  of  data  would  provide  a  true  situational  awareness  of 
the  overall  program. 

TR  delivery  metrics 

Standardized  metrics  could  be  created  to  measure  the  amount  of 
time  creating  a  TR  requires  from  the  end  of  testing  until  delivery 
to  the  customer,  with  authorized  users  able  to  view  where  the 

TR  is  within  the  process  at  any  given  time. 

Search 

The  IWE  should  allow  authorized  users  to  search  for  and 
retrieve,  security  trimmed  results,  artifacts  through  user-friendly 
searches.  Made  possible  through  the  indexing  of  all  data 
regardless  of  whether  the  data  is  structure  or  unstructured  and 
whether  it  has  metadata  or  tagging  associated  or  not. 

Table  20.  Summary  of  Changes  for  Cloud  T&E  Process 
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V.  CONCLUSION  AND  FUTURE  RESEARCH 


A.  KEY  FINDINGS  AND  RECOMMENDATIONS 

In  this  thesis,  we  analyzed  the  existing  workflow  processes  that  Army  T&E  users 
follow  during  the  execution  of  a  T&E  program.  We  then  assessed  how  cloud  computing 
can  be  used  to  streamline  these  workflows. 

In  the  course  of  documenting  the  Army  T&E  workflow  processes,  we  focused  our 
attention  on  communications,  and  collaborations  within  the  enterprise.  This  included 
communications  starting  with  a  request  for  test  services,  followed  by  scheduling  a  test 
and  compilation  and  delivery  of  a  final  test  report.  The  documentation  consisted  of 
scenarios,  activity  diagrams,  and  collaboration  diagrams  for  nine  use  cases.  During  the 
analysis  of  the  current  system  use  cases,  we  determined  that  the  current  system  relies 
heavily  on  e-mail  as  the  primary  means  of  communication  and  file  transport.  This  can 
result  in  delays  in  the  delivery  of  infonnation  to  the  PM  as  infonnation  is  relayed  from 
the  PM  down  through  the  chain  of  command  to  the  TE  and  then  back  up  the  chain  of 
command  to  the  PM.  These  delays  can  potentially  affect  decisions  made  by  the  PM  and 
senior  leadership. 

The  use  cases,  activity,  and  collaboration  diagrams  mentioned  above  were  then 
reworked  to  show  how  the  processes  could  be  improved  upon  to  leverage  the 
communication  and  collaboration  capabilities  afforded  by  cloud  computing.  We 
determined  that  it  is  possible  to  streamline  several  of  the  current  processes,  which  were 
based  on  manual  or  e-mail  collaborations.  Specifically  the  Program  Management  process, 
Test  Report  Generation  process,  and  the  Document  Collaboration  process  would  all 
benefit  from  a  move  to  the  cloud.  These  three  processes  and  potential  improvements  are 
to  some  extent  generalizable  beyond  the  T&E  domain.  For  instance  the  Document 
Collaboration  process  is  applicable  within  every  DoD  environment  that  edits  documents 
and  the  collaborations  within  the  Program  Management  process  would  be  applicable 
across  multiple  services,  that  is,  beyond  just  the  Army. 
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The  streamlining  could  be  realized  through  the  use  of  cloud-based  collaboration 
tools  such  as:  online  document  editors,  instant  messaging,  threaded  message  boards, 
wikis,  blogs,  tags,  status  updates,  news,  hot  topics,  tasks,  and  RSS  feeds.  These 
collaboration  tools,  along  with  all  collected  data  associated  with  a  program,  would  be 
accessible  through  an  IWE.  The  IWE  would  provide  a  central  location  for  storing, 
reducing,  and  collaborating  on  all  information  related  to  a  program  replacing  e-mail  as 
the  primary  means  of  collaboration  and  file  transport.  Where  possible  the  IWE  would 
interface  with,  and  display  information  from,  legacy  ATEC  Enterprise  systems  such  as 
ADSS  and  SOFIMS. 

The  establishment  of  an  IWE  would  greatly  assist  in  the  timely  delivery  of 
information  to  the  PM  and  across  the  entire  enterprise.  The  IWE  would  provide  a  mash- 
up  of  scheduling,  financial,  and  other  programmatic  information  that  the  PM,  or  anyone 
else  with  proper  access  rights,  could  access  and  pull  information  from  on  an  as  needed 
basis.  Although  the  PM’s  workflow  did  not  change  substantially,  by  freeing  the  PM  up 
from  the  timely  manual  tasks  of  collecting  information  about  programs  undergoing  T&E, 
he  or  she  can  better  use  that  freed  up  time.  The  effects  of  improving  and  modifying  the 
PM’s  workflow  process  will  be  propagated  to  the  related  workflow  processes  of  others,  at 
a  minimum  the  units  of  the  enterprise  that  support  the  PM  and  ultimately  the  customers 
and  other  stakeholders. 

Although  this  thesis  focused  solely  on  the  Army  T&E  processes,  it  is  likely  that 
any  improvements  obtained  by  moving  the  Army  processes  to  the  cloud  would  also  be 
applicable  for  Joint  T&E  programs.  Joint  system  T&E  programs,  meaning  systems  that 
are  utilized  across  multiple  services  such  as  the  Ballistic  Missile  Defense  System 
(BDMS),  are  faced  with  the  same  type  of  communications  and  collaborations  as  the 
Army  processes.  However,  joint  programs  also  have  to  manually  maneuver  through  each 
service’s  unique  T&E  workflow  process.  The  IWE  could  serve  as  an  interface  to  the  data 
and  applications  in  the  cloud,  with  the  workflow  details  of  the  different  services  being 
transparent  to  the  user. 
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We  identified  some  specific  ways  in  which  to  support  the  workflow  of  the  PM 
such  as  enabling  the  PM  to  create  a  composite  view  of  the  schedule  for  the  entire 
program,  regardless  of  how  many  test  centers  are  involved  in  the  program.  The  advantage 
to  the  PM  of  having  access  to  such  a  view  of  the  data  is  that  he  or  she  can  learn  about 
slippage  at  one  range  that  may  affect  another  test  center’s  long  range  schedule.  At  the 
other  end  of  the  spectrum,  having  all  test  data  for  a  program  stored,  accessed,  and 
processed  through  the  IWE  would  allow  the  ISP  to  automatically  secure  the  data  based  on 
the  POC  listed  in  the  schedule  tool.  This  would  remove  the  burden  from  the  TE  while 
also  reducing  the  risk  of  a  competing  contractor  stumbling  across  a  competitor’s  data. 

The  success  or  failure  of  the  IWE  concept  will  rely  heavily  on  its  ability  of  to 
provide  Google-like  search  capabilities.  Users  should  be  able  to  search  on  both  structured 
and  unstructured  data,  without  requiring  the  user  posing  the  query  to  add  metadata  or  tag 
the  data.  Within  the  scenarios  of  this  thesis  we  assumed  that  the  user  would  tag  the  data 
upon  either  initial  collection  or  at  any  major  milestone  such  as  changing  the  status  of  a 
document  from  draft  to  a  final  release.  This  assumption  may  be  unrealistic  within  a  real- 
world  setting  in  which  the  manual  process  is  unreliable.  If  the  tagging  of  the  data  cannot 
be  automated  in  some  fashion,  such  as  by  pulling  the  scheduling  tool  metadata  in,  then 
the  crux  of  the  IWE  will  always  be  limited  by  its  search  and  index  functionality. 

B.  CONCLUDING  REMARKS 

If  the  history  of  technology  repeats  itself,  those  prepared  for  IT  change  will  be 
better  positioned  to  take  full  advantage  of  new  opportunities  (“Cloud  computing: 
Paradigm  shift  or  just  hype,”  2008).  Cloud  computing  is  a  disruptive  technology  whose 
implementation  will  require  change  across  all  levels  of  DoD.  It  will  require  technical 
training  and  a  cultural  shift  in  how  DoD  senior  leadership,  program  management,  end 
users,  customers,  suppliers,  and  especially  IT  professionals  think  about  IT  resources.  This 
shift  will  require  changes  in  all  aspects  of  the  acquisition  of  IT.  In  addition  to  the 
technical  challenges,  there  will  be  challenges  in  aligning  the  corporate  culture  with  the 
new  workflows  and  associated  means  of  communication  and  collaboration. 
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The  cultural  issues  surrounding  a  user’s  trust  that  the  network  will  be  there  when 
it  is  needed  will  be  one  of  the  most  difficult  challenges  to  overcome.  In  DoD,  as  at  many 
organizations,  if  you  own  it,  you  control  it  (Zyskowski,  2010).  As  an  IT  professional,  if 
you  cannot  go  into  a  cold  server  room  and  see  a  row  full  of  blinking  lights,  hear  fans 
humming,  and  have  the  ability  to  ‘hug  the  server’  how  can  you  trust  that  it  will  be  there 
when  your  users  and  senior  leadership  requires  it?  As  a  senior  leader,  how  do  you 
overcome  the  fear  associated  with  no  longer  having  someone  in  your  chain  of  command 
who  you  can  ‘reach  out  and  touch’  24/7  if  a  business  critical  capability  goes  offline?  In 
the  current  atmosphere,  most  critical  servers  and  applications  reside  locally,  and  if  a 
capability  fails,  fixing  it  is  merely  a  matter  of  dedicating  enough  man  hours  and  hardware 
to  repair  the  system.  In  a  public  or  private  cloud  environment,  you  are  potentially  placing 
cloud-services  providers,  outside  of  your  immediate  control,  directly  into  your  IT 
department’s  critical  path  for  keeping  systems  available  and  operating  correctly. 


Figure  48.  Response  to  change  from  (Koch,  2004) 

While  change  is  natural  and  good,  the  typical  first  reaction  to  change  is  resistance 
that  comes  from  a  fear  of  the  unknown  or  an  expectation  of  loss  (Figure  48).  Resistance 
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to  cloud  computing  will  be  met,  and  for  DoD’s  transfonnation  to  succeed,  this  resistance 
must  be  overcome.  In  general,  fighting  human  nature  is  an  uphill  battle  that  eventually 
results  in  failure.  However,  by  working  to  educate  stakeholders  on  what  cloud  computing 
is,  defining  what  will  change,  why  it  needs  to  change  and  getting  a  mutual  understanding 
from  key  stakeholders,  hopefully  their  resistance  to  change  can  be  overcome. 

C.  FUTURE  WORK 

While  cloud  computing  is  still  in  its  infancy,  this  research  has  shown  that  it  does 
bear  promise  cut  the  cost  of  delivering  IT  services  to  the  DoD  community,  other  things 
being  equal.  However,  there  are  lots  of  open  research  questions,  some  of  which  we 
mention  below: 

1.  Near  Term 

a.  Network  Bandwidth  Measurement 

Network  speed  between  the  end  user  and  the  cloud  will  be  a  potential 
stumbling  block.  Many  TCs  are  in  remote  locations  and  have  limited  network  bandwidth. 
The  fiber  infrastructure  into  these  TCs  will  likely  need  to  be  improved  prior  to  a  large- 
scale  move  to  a  cloud  environment.  The  rise  of  the  use  of  Voice  Over  IP  (VOIP) 
telephones  will  also  increase  the  amount  of  network  traffic.  This  could  have  a  negative 
impact  to  moving  large  amounts  of  data  into  the  cloud  or  trying  to  use  cloud-based  data- 
reduction  tools  via  the  network.  Additional  investigations  should  focus  on  gathering 
technical  metrics  and  measuring  the  bandwidth  of  networks  between  various  DoD 
locations  and  the  as  yet  to  be  announced  DoD  APC  cloud  locations.  These  studies  should 
focus  on  measuring  available  bandwidth  over  short  and  long  distances  to  identify 
potential  communication  bottlenecks. 

b.  Additional  Use  Cases 

Further  investigations  should  be  conducted  to  document  user  requirements 
and  use  cases  beyond  those  identified  in  this  thesis.  This  research  should  be  conducted 
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within  both  the  NIPRNet  and  DREN,  as  these  two  environments  have  different  missions 
and  requirements.  Research  should  also  be  conducted  on  the  secure  side  (e.g.,  SIPRNet 
and  SDREN  environments). 

c.  IT  Technologist’s  Role  Within  the  Cloud 

Research  should  begin  to  identify  the  delta  of  a  typical  IT  Server/Network 
administrator’s  daily  duties  today  versus  after  their  assets  are  deployed  into  a  cloud 
environment.  Vendors  and  cloud  proponents  claim  that  moving  to  the  cloud  will  lead  to  a 
fundamental  shift  in  IT  goals  enabling  IT  professionals  to  spend  less  time  and  effort  on 
the  data  center  and  help  desk  activities.  This  freed  up  time  theoretically  would  allow  IT 
professionals  to  spend  more  time  working  with  end  users  on  business  innovation  and 
improvement  projects. 

d.  Pathfinders  and  Pilot  Programs 

Pathfinder  experiments  should  be  created  to  test  out  cloud  computing 
within  DoD’s  multiple  environments  (e.g.,  NIPRNet,  SIPRNet,  DREN,  SDREN).  The 
pathfinders  would  allow  researchers  to  evaluate  the  speed,  security,  usability,  SLAs,  and 
other  aspects  of  potential  cloud-based  solutions. 

After  successful  pathfinder  experiments,  multiple  pilot  programs  should 
be  created  at  locations  geographically  close  to  DoD  APC  cloud  locations,  as  well  as 
remote  locations.  The  pilot  programs  should  focus  on  moving  small  organizations  into 
the  cloud.  This  will  assist  DoD  in  getting  any  kinks  out  of  the  process  prior  to  widespread 
adoption.  The  geographically  close  pilot  could  be  used  to  demonstrate  what  theoretically 
should  be  a  best-case  scenario  from  a  performance  standpoint,  while  the  remote  location 
pilot  could  be  used  to  identify  issues  that  are  likely  to  arise  during  widespread  adoption. 

The  pilot  programs  would  also  provide  an  opportunity  to  evaluate  the 
change  in  IT  staffs  responsibilities  and  duties.  The  upcoming  Army  APC2 
implementation  would  be  a  good  opportunity  for  research  to  begin  from  the  ground  floor 
and  collect  the  metrics  needed  for  the  studies  suggested  above. 
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e. 


Security  Concerns 


Storing  all  test  data  within  the  same  repository,  whether  physical  or 
logical,  and  accessing  through  a  common  interface  will  raise  issues  with  classification 
through  aggregation.  Research  should  address,  among  other  things,  ways  to  mitigate 
information  leakage  and  the  establishment  of  redaction  procedures. 

Research  should  also  begin  to  identify  whether  current  architectures  are 
adequate  for  building  trusted  clouds,  or  if  new  architectures  are  needed.  Whether  current 
OS  and  application  security  approaches  will  scale  properly  to  a  cloud  computing 
environment,  or  if  new  approaches  will  need  to  be  taken  to  provide  a  trusted  OS — such  as 
the  efforts  currently  underway  within  NPS  and  as  described  in  the  DoD  IAnewsletter 
article  titled  “Establishing  Trust  in  Cloud  Computing”.  (Dinolt  &  Michael,  2010) 

f  Social  Networking  Tools 

Further  research  should  occur  around  the  usage  of  social  networking  tools 
within  DoD.  This  thesis  only  briefly  touched  on  the  potential  usage  of  social  networking 
tools  within  T&E.  Current  social  networking  tools  should  be  evaluated  for  their 
applicability  to  advancing  the  DoD  mission. 

2.  Long  Term 

a.  Data  Tagging,  Indexing,  and  Searching 

Further  research  should  occur  in  the  realm  of  automated  tagging  of  data. 

The  success  of  the  ICW  is  predicted  to  rely,  in  some  degree,  upon  the  automated  tagging 
of  all  data  artifacts  with  metadata  and  the  ability  to  efficiently  search  and  index  structured 
and  unstructured  data. 

b.  Tactical  Applicability 

Connecting  multiple  information  sources  together  in  a  meaningful  way  to 

provide  commanders  and  warfighters  with  a  more  complete  picture  is  critical  to  making 

better  decisions.  Information  should  be  just  as  available  at  the  edge  of  the  battlefield, 

senior  leaders  at  headquarters,  and  TEs  at  the  T&E  ranges.  Research  should  also  be 
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undertaken  on  assessing  the  applicability  of  cloud  computing  to  the  tactical  environment. 
Use  cases  should  also  be  gathered  for  applicability  to  the  warfighter  on,  or  near,  the 
frontlines.  Some  of  the  current  research  efforts  include:  “Cloud  Computing  for  Large- 
Scale  Weapon  Systems”  (Foster  et  ah,  2010b)  and  “The  Cloud  and  its  Implications  to 
Naval  Warfare”  (Hurlburt,  2010). 
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